🥫 The simple, fast, and modern web scraping library

Describe the bug

When I create a soup object...

To Reproduce

Calling .text returns an empty string:

from gazpacho import Soup

html = """<p>&pound;682m</p>"""

soup = Soup(html)
print(soup.text)
''

Expected behavior

Should output:

print(soup.text)
'£682m'

Environment:

• OS: macOS
• Version: 1.1

Additional context

Inspired by this S/O question

Describe the bug When trying to get text from a tag, gazpacho returns empty string

from gazpacho import Soup

html  = '<a href="/Sorasful?source=gig_cards&referrer_gig_slug=edit-mixing-and-mastering&ref_ctx_id=42d34014-b499-46fa-a1d3-04318b12fecc" rel="nofollow noopener noreferrer" target="_self"><span>by </span>Sorasful</a>'

soup = Soup(html)

print(soup.text)
# prints nothing

print(soup.find('a').text)
# prints "by"

The default auto behavior of .find() doesn't work for me, because it means I can't trust my code not to start throwing errors if the page I am scraping adds another matching element, or drops the number of elements down to one (triggering a change in return type).

I know I can do this:

div = soup.find("div", mode="first")
# Or this:
divs = soup.find("div", mode="all")

But having function parameters that change the return type is still a bit weird - not great for code hinting and suchlike.

Changing how .find() works would be a backwards incompatible change, which isn't good now that you're past the 1.0 release. I suggest adding two new methods instead:

div = soup.first("div") # Returns a single element
# Or:
divs = soup.all("div") # Returns a list of elements

This would be consistent with your existing API design (promoting the mode arguments to first class method names) and could be implemented without breaking existing code.

Is your feature request related to a problem? Please describe. It's great to be able to run find and then find within the initial result, but it seems more readable to be able to find based on CSS selectors.

Describe the solution you'd like

selector = '.foo img.bar'
soup.select(selector) # this would return any img item with the class "bar" inside of an object with the class "foo"

Is your feature request related to a problem? Please describe. Right now it's hard to reason about the behaviour of the find method. If it finds one element it will return a Soup object, if it finds more than one it will return a list of Soup objects.

Describe the solution you'd like Separate find into a find method and find_one method.

Describe alternatives you've considered Keep it and YOLO?

Additional context Conversation with Michael Kennedy:

If I were designing the api, i'd have that always return a List[Node] (or whatever the class is). Then add two methods:

• find() -> List[Node]
• find_one() -> Optional[Node]
• one() -> Node (exception if the there are zero or two or more nodes)

Describe the bug

Soup can handle and format matched tags no problem:

from gazpacho import Soup
html = """<ul><li>Item 1</li><li>Item 2</li></ul>"""
Soup(html)

Which correctly formats to:

<ul>
  <li>Item 1</li>
  <li>Item 2</li>
</ul>

But it can't handle void tags (like img)...

To Reproduce

For example, this bit of html:

html = """<ul><li>Item 1</li><li>Item 2</li></ul><img src="image.png">"""
Soup(html)

Will fail to format on print:

<ul><li>Item 1</li><li>Item 2</li></ul><img src="image.png">

Expected behavior

Ideally Soup formats it as:

<ul>
  <li>Item 1</li>
  <li>Item 2</li>
</ul>
<img src="image.png">

Environment:

• OS: macOS
• Version: 1.1

Additional context

The problem has to do with the underlying parseString function unable to handle void tags:

from xml.dom.minidom import parseString as string_to_dom
string_to_dom(html)

Possible solution, turn void tags into self-closing tags on input, and the transform them back to void tags on print....

$ git tag v0.7.2 && git push --tags 🎉 🎈

I really like this project. I think that adding releases to the repository can help the project grow in popularity. I'd like to see that!

Is your feature request related to a problem? Please describe.

It might be nice if gazpacho had the ability to rotate/fake a user agent

Describe the solution you'd like

Sort of like this but more primitive. (Importantly gazpacho does not want to take on any dependencies)

Additional context

Right now gazpacho just spoofs the latest Firefox User Agent

Describe the bug Right now match has an ability to be strict. This functionality is presently not enable for find.

To Reproduce Code to reproduce the behaviour:

from gazpacho import Soup, match

match({'foo': 'bar'}, {'foo': 'bar baz'})
# True

match({'foo': 'bar'}, {'foo': 'bar baz'}, strict=True)
# False

Expected behavior The find method should be forgiving (partial match) to protect ease of use, and maintain backwards compatibility, but there should be an argument to enable strict/exact matching that piggybacks on match

Environment:

• OS: macOS
• Version: 0.7.2

Is your feature request related to a problem? Please describe. I would like try to a .children() method in the Soup object that can list all the child elements of the Soup object.

Describe the solution you'd like I would make a regex pattern to match each inner element and return a list of Soup() objects with those elements. I might also try to make an option for recurse or not.

Describe alternatives you've considered All that I can think of is doing the same thing mentioned above in the scraping code

Additional context None

Is your feature request related to a problem? Please describe. Improve the .github issue template

Describe the solution you'd like I would like a better issue and feature request template in the .github folder. The format I would like is the bolded headings to become proper sections, and the help line below them comments.

Describe alternatives you've considered None

Additional context What I would like is instead of:

---
name: Bug report
about: Create a report to help gazpacho improve
title: ''
labels: ''
assignees: ''
---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Code to reproduce the behaviour:

```python

\```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Environment:**
 - OS: [macOS, Linux, Windows]
 - Version: [e.g. 0.8.1]

**Additional context**
Add any other context about the problem here.

It should be something like:

---
name: Bug report
about: Create a report to help gazpacho improve
title: ''
labels: ''
assignees: ''
---

## Describe the bug
<!-- A clear and concise description of what the bug is. -->

## To Reproduce
<!-- Code to reproduce the behaviour: -->

```python
# code
\```

## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->

**Environment:**
 - OS: [macOS, Linux, Windows]
 - Version: [e.g. 0.8.1]

## Additional context
<!-- Add any other context about the problem here. Delete this section if not applicable -->

Or something like this

Need support for dynamic text rendering...

Need a method that triggers the Javascript on a page to fire (see https://github.com/psf/requests-html, r.html.render()).

Describe the bug

Can't parse some entries, there are 40 entries for every page, but some are not being parsed correctly.

Steps to reproduce the issue

from gazpacho import get, Soup

for i in range(1, 15):
    link = f'https://1337x.to/category-search/aladdin/Movies/{i}/'
    html = get(link)
    soup = Soup(html)
    body = soup.find("tbody")

    # extracting all the entries in the body,
    # there are 40 entries for every page, the last one can have less,
    entries = body.find("tr", mode='all')[::-1]

    # but for some pages it can't retrives all the entries from some reason
    print(f'{len(entries)} entries -> {link}')

Expected behavior

See 40 entries for every page

Environment:

Arch Linux - 5.13.10-arch1-1 Python - 3.9.6 Gazpacho - 1.1

Describe the bug

Using soup.find on particular website(s) returns entire html instead of the matching tag(s)

Steps to reproduce the issue

Look for ul tag with attribute class="cves" (<ul class="cves">) on https://mariadb.com/kb/en/security/

from gazpacho import get, Soup
endpoint = "https://mariadb.com/kb/en/security/"
html_dump = Soup.get(endpoint)
sample = html_dump.find('ul', attrs={'class': 'cves'}, mode='all')

sample contains the contents of an entire html

Expected behavior

sample should contain the contents of the tag <ul class "cves">, which in this case would be rows of <li>-s, listing the CVEs and corresponding fixed version in MariaDB, something like:

<ul class="cves">
  <li>..</li>
  ...
  <li>..</li>
</ul>

Environment:

• OS: Ubuntu Linux 18.04
• Version: gazpacho 1.1, python 3.6.9

Additional information

Using BeautifulSoup on the same html_dump did get the job done, although the <li>-tags are weirdly nested together.

from bs4 import BeautifulSoup
# html_dump from above Soup.get(endpoint)
bs_soup = BeautifulSoup(html_dump.html, 'html.parser')
ul_cves = bs_soup.find_all('ul','cves')

ul_cves contain strangely nested <li>-s, from which it was still possible to extract the rows of <li>-s I was looking for.

<ul class="cves">
  <li>
    <li>
    ...
  </li></li>
</ul>

Thank you for your nice project!

Please add an argument encoding to decode that does not utf-8 encoded pages. https://github.com/maxhumber/gazpacho/blob/ecd53aff4e3d8bdf9eaaea4e0244a75cbabf6259/gazpacho/get.py#L51

I tried EUC-KR encoded page and got an error message.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 95: invalid start byte

find changes the content of attrs

When using the find method on a Soup object, the content of attrs is overwritten by the parameter attrs in find.

Steps to reproduce the issue

Try the following:

from gazpacho import Soup

div = Soup("<div id='my_id' />").find("div")
print(div.attrs)
div.find("span", {"id": "invalid_id"})
print(div.attrs)

The expected output will be the following, because we twice print the attributes of a:

{'id': 'my_id'}
{'id': 'my_id'}

But instead you actually receive:

{'id': 'my_id'}
{'id': 'invalid_id'}

which is wrong.

Environment:

• OS: Linux
• Version: 1.1

My current workaround is to save the attributes before I execute find.

Describe the bug

Hi,

There was a pull request (https://github.com/maxhumber/gazpacho/pull/48) to add whl publishing but it appears to have been lost somewhere in a merge on October 31st, 2020. (https://github.com/maxhumber/gazpacho/compare/v1.1...master). Therefore, no wheels have been published for 1.1.

This causes the installation error on my system that the PR was meant to address.

Expected behavior

Install gazpacho with a wheel, not a tar.gz;. Please re-add the whl publishing.

Environment:

• OS: Windows 10