I noticed that the graph created use the filepath, and I want to choose the frontmatter title or the filename instead. How can I do that ?

Graphic reference : Generated using pyvis

Running the following on my vault:

import obsidiantools.api as ot
vault = ot.Vault(Path("/path/to/a/vault").connect()

Results in:

# --->8--- Irrelevant frames omitted --->8---

~/.cache/pypoetry/virtualenvs/knowledgebase-scripts-Fe_uWe_V-py3.9/lib/python3.9/site-packages/obsidiantools/md_utils.py in _get_ascii_plaintext_from_md_file(filepath)
    190     html = _get_html_from_md_file(filepath)
    191     # strip out front matter (if any):
--> 192     html = _remove_front_matter(html)
    193     return _get_ascii_plaintext_from_html(html)
    194 

~/.cache/pypoetry/virtualenvs/knowledgebase-scripts-Fe_uWe_V-py3.9/lib/python3.9/site-packages/obsidiantools/md_utils.py in _remove_front_matter(html)
    201     if hr_content:
    202         # wipe out content from first hr (the front matter)
--> 203         for fm_detail in hr_content.find_next("p"):
    204             fm_detail.extract()
    205         # then wipe all hr elements

TypeError: 'NoneType' object is not iterable

A quick roundtrip in a debugger shows this happens with at least:

1. Notes containing an hr (---) but no YAML frontmatter.
2. Notes containing only frontmatter, no body.

Hi.

Every markdown file is being opened & read a total of 8 times in normal connect & gather flow. Might make sense to model a note as a class and have it load its own data once.

Hi,

I can successfully view my vault file index in Windows. If I then try to filter the list by subdirectory I can successfully list notes in the root and in the 'docs' folders. If I filter by the name of a lower subdirectory using a Windows filepath the returned list is empty.

For example, my file index includes the following list items:

{'README': WindowsPath('README.md'),
 'index': WindowsPath('docs/index.md'),
 'Quotations': WindowsPath('docs/Quotations.md'),
 'Creative Commons': WindowsPath('docs/Concepts/Creative Commons.md'),
 'Crowdsourcing': WindowsPath('docs/Concepts/Crowdsourcing.md'),
 'Data Format': WindowsPath('docs/Concepts/Data Format.md'),
 'Data Model': WindowsPath('docs/Concepts/Data Model.md'),
 'Data Sovereignty': WindowsPath('docs/Concepts/Data Sovereignty.md'),
}

Based on the obsidiantools-demo I would expect to be able to list all the markdown files in the 'Concepts' folder using the following call:

(otools.Vault(vault_dir, include_subdirs=['docs/Concepts'], include_root=False) .file_index)

Instead the returned object is empty {}.

Reversing the slash to create a linux path resolves the issue:

(otools.Vault(vault_dir, include_subdirs=['docs\Concepts'], include_root=False)
 .file_index)

Returns:

{'Creative Commons': WindowsPath('docs/Concepts/Creative Commons.md'),
 'Crowdsourcing': WindowsPath('docs/Concepts/Crowdsourcing.md'),
 'Data Format': WindowsPath('docs/Concepts/Data Format.md'),
 'Data Model': WindowsPath('docs/Concepts/Data Model.md'),
 'Data Sovereignty': WindowsPath('docs/Concepts/Data Sovereignty.md')}

Ideally this would be resolved by the obsidiantools package rather than the user. Alternatively suggest updating the documentation.

Hi, I'm testing out the package and I'm getting an error when I try to connect via Jupyter notebook in Windows 10: vault = otools.Vault(vault_dir).connect().gather().

I'm receiving the following UnicodeDecodeError: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1400: character maps to <undefined>.

Assuming the filepath and connection are working I'm unclear whether it is a problem I can correct in Obsidian or if it is a problem with the parser used by obsidiantools. Can you advise how I can resolve the issue?

For reference the stack trace is as follows:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_19592/2568229718.py in <module>
----> 1 vault = otools.Vault(vault_dir).connect().gather()
      2 print(f"Connected?: {vault.is_connected}")
      3 print(f"Gathered?:  {vault.is_gathered}")

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\api.py in connect(self)
    199         if not self._is_connected:
    200             # default graph to mirror Obsidian's link counts
--> 201             wiki_link_map = self._get_wikilinks_index()
    202             G = nx.MultiDiGraph(wiki_link_map)
    203             self._graph = G

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\api.py in _get_wikilinks_index(self)
    438         where k is the md filename
    439         and v is list of ALL wikilinks found in k"""
--> 440         return {k: get_wikilinks(self._dirpath / v)
    441                 for k, v in self._file_index.items()}
    442 

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\api.py in <dictcomp>(.0)
    438         where k is the md filename
    439         and v is list of ALL wikilinks found in k"""
--> 440         return {k: get_wikilinks(self._dirpath / v)
    441                 for k, v in self._file_index.items()}
    442 

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\md_utils.py in get_wikilinks(filepath)
     92         list of strings
     93     """
---> 94     plaintext = _get_ascii_plaintext_from_md_file(filepath, remove_code=True)
     95 
     96     wikilinks = _get_all_wikilinks_from_html_content(

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\md_utils.py in _get_ascii_plaintext_from_md_file(filepath, remove_code)
    265     """md file -> html -> ASCII plaintext"""
    266     # strip out front matter (if any):
--> 267     html = _get_html_from_md_file(filepath)
    268     if remove_code:
    269         html = _remove_code(html)

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\md_utils.py in _get_html_from_md_file(filepath)
    251 def _get_html_from_md_file(filepath):
    252     """md file -> html (without front matter)"""
--> 253     _, content = _get_md_front_matter_and_content(filepath)
    254     return markdown.markdown(content, output_format='html')
    255 

~\anaconda3\envs\Obsidian_Tools\lib\site-packages\obsidiantools\md_utils.py in _get_md_front_matter_and_content(filepath)
    242     with open(filepath) as f:
    243         try:
--> 244             front_matter, content = frontmatter.parse(f.read())
    245         except yaml.scanner.ScannerError:
    246             # for invalid YAML, return the whole file as content:

~\anaconda3\envs\Obsidian_Tools\lib\encodings\cp1252.py in decode(self, input, final)
     21 class IncrementalDecoder(codecs.IncrementalDecoder):
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24 
     25 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1400: character maps to <undefined>

It seems that obsidiantools does not track non-markdown files (pictures, etc). As a consequence, vault.nonexistent_notes list all references to such files. I suggest to include non-markdown files in the graph as well.

And, on a related note, vault.nonexistent_notes wrongly includes notes that are referenced with extension. For example, a reference of the form [[note.md]] leads to note.md being listed as non-existent even if the file note.md exists.

in case of a malformed frontmatter in a document an exception is raised and not handled.

• error can be handled in https://github.com/mfarragher/obsidiantools/blob/ddd78669ef27346fdb4b13cdc956a6f8c00e98f4/obsidiantools/md_utils.py#L259
• adding the following solves the problem (allthough the specific error should be named)

except:
    print("problem with file ", filepath)

Hello. Thanks for this library

I have tried this with 3.9 as suggested. But I'm getting an error straight away on gather.

File "/home/steve/.pyenv/versions/obsidian-python3.9.0/lib/python3.9/site-packages/obsidiantools/md_utils.py", line 29, in get_md_relpaths_from_dir
    for p in glob(str(dir_path / '**/*.md'), recursive=True)]
TypeError: unsupported operand type(s) for /: 'str' and 'str'

I have to change the /' to a + in the line in get_md_relpaths_from_dir() to get the globbing working!

change from

return [Path(p).relative_to(dir_path)
for p in glob(str(dir_path / '**/*.md'), recursive=True)]

to

return [Path(p).relative_to(dir_path)
for p in glob(str(dir_path + '**/*.md'), recursive=True)]

As a placeholder. I think code blocks should be ignored for tags? What do you think?

 "file_tags": [
        "meta",
        "idea",
        "shower-thought",
        "to-digest",
        "shower-thought",
        "introduction",
        "shower-thought\"",
        "guru\"",
        "shroedinger-uncertain\"",
        "floating-point-error\"",
        "socratic\""
      ]

[[Foo]] and [[Bar.md]] will both be related to note 'Foo' in the knowledge graph.

Currently, wikilinks getters will extract the wikilinks as 'Foo' and 'Bar.md'. The expected behaviour of getters to reflect Obsidian's behaviour is 'Foo' and 'Bar' respectively.

For one of my notes with a mix of tables, LaTeX, lists & code blocks, there is a lot of text from the note that isn't captured in source_text_index, but is kept in the HTML. This suggests some parsing issues with how html2text is configured.

Whole paragraph blocks & headers can be completely missing.

This starts to happen after a table with LaTeX. Anything in body text (<p>) afterwards is missing, yet it keeps all the remaining LaTeX (even the stuff in tables).

Perhaps it doesn't like MathJax? Maybe wiping out a few tags from HTML, for the source_text functionality, before it gets processed by html2text could make the output smoother in this case.

Need to think more about:

• What HTML tags are not necessary for source_text?
  • LaTeX is one aspect to remove if causing problems. Keep as much as possible for html2text to handle (including URLs, images, etc.). Anything more opinionated (e.g. do we want strikethrough text or not) would be better covered in readable_text.
  • May involve another Markdown class if switching off markdown extensions, more functions to do this specific HTML generation, etc.
• A test case from reduced format of my note

Hey, for https://github.com/louis030195/obsidian-ava, I'm trying to implement increment refresh of the state of the vault.

Concretely, I build sentence embeddings of the whole vault and would like to re-compute embeddings every time a note is updated/deleted/created.

Do you see any way of doing this incrementally rather than reloading the vault and recomputing everything every time? (It takes ~1 min on mps device on my 500k words vault)

Ideally, I'd see maybe an API that let me listen to vault changes with callback(s) in this library?

Thanks 🚀😃