Beautiful Soup - Crawl Wiki Page

I'm trying to crawl a list on a wiki page "" taking the titles / descriptions of each nautical term, my first issue was correctly dealing with the lists within the descriptions like so:

import requests
from bs4 import BeautifulSoup

url = ''
page = requests.get(url)

get_title = []
get_desc = []
corrected_desc = []
output = ''

if page.status_code == 200:
    soup = BeautifulSoup(page.text, 'html.parser')
    get_title = soup.find_all('dt', class_='glossary')
    get_desc = soup.find_all('dd', class_='glossary')

    for i in get_desc:
        first_char = i.get_text()[:1]
        second_char = i.get_text()[1:2]

        if (first_char.isnumeric() and second_char == '.'):
            if(first_char == '1' and output):
                output = ''
                output += '{} '.format(i.get_text())
                output += '{} '.format(i.get_text())

        if output:
            output = ''
    print('failed to get the page!')

print(str(len(get_title)) + ' - ' + str(len(corrected_desc)))
zipped = zip(get_title, corrected_desc)

for j in zipped:
    output = '{}, {}\n'.format(j[0].get_text(), j[1].strip())
    with open('test.txt', "a", encoding='utf-8') as myfile:

But I can't seem to work out how to handle a description with both a list and a sentence.

Edit: The output i'm looking for is:

"Title", "Description"
"Title", "Description"
"Title", "Description"
"Title", "Description"

but i'm not sure how to go about adapting my code to handle cases where the description is a list + a sentence.

1 answer

  • answered 2018-04-14 15:42 Keyur Potdar

    All the titles are inside the <dt> tag, and the descriptions are inside the <dd> tag. So, the first step is to find all those tags. It can be done using soup.find_all(['dt', 'dd']). Then, loop over these tags and check whether the tag is dt or dd using if == 'dt'. If the tag is dd append its contents to the description variable, else print the current contents of the variables.

    Complete code:

    import requests
    from bs4 import BeautifulSoup
    r = requests.get('')
    soup = BeautifulSoup(r.text, 'lxml')
    curr_title, curr_description = '', ''
    for tag in soup.find_all(['dt', 'dd']):
        if == 'dt':
            if curr_title:
                print('{}: {}'.format(curr_title, curr_description))
                curr_description = ''
            curr_title = tag.text.strip()
            curr_description = ' '.join((curr_description, tag.text.strip()))

    Partial output:

    A-back:  A foresail when against the wind, used when tacking to help the vessel turn.[1]
    Abaft:  Toward the stern, relative to some object ("abaft the fore hatch").
    Abaft the beam:  Further aft than the beam: a relative bearing of greater than 90 degrees from the bow: "two points abaft the beam, starboard side". That would describe "an object lying 22.5 degrees toward the rear of the ship, as measured clockwise from a perpendicular line from the right side, center, of the ship, toward the horizon."[2]
    Abandon ship!:  An imperative to leave the vessel immediately, usually in the face of some imminent overwhelming danger.[3] It is an order issued by the Master or a delegated person in command. (It must be a verbal order). It is usually the last resort after all other mitigating actions have failed or become impossible, and destruction or loss of the ship is imminent; and customarily followed by a command to "man the lifeboats" or life rafts.[3][4]
    Abeam:  On the beam, a relative bearing at right angles to the ship's keel.[5]
    Able seaman:  Also able-bodied seaman. A merchant seaman qualified to perform all routine duties, or a junior rank in some navies.
    Aboard:  On or in a vessel. Synonymous with "on board." (See also close aboard.)
    About:  "To go about is to change the course of a ship by tacking. Ready about, or boutship, is the order to prepare for tacking."[6]
    Above board:  On or above the deck, in plain view, not hiding anything. Pirates would hide their crews below decks, thereby creating the false impression that an encounter with another ship was a casual matter of chance.[7]
    Above-water hull:  The hull section of a vessel above the waterline, the visible part of a ship. Also, topsides.
    Absentee pennant:  Special pennant flown to indicate absence of commanding officer, admiral, his chief of staff, or officer whose flag is flying (division, squadron, or flotilla commander).
    Absolute bearing:  The bearing of an object in relation to north. Either true bearing, using the geographical or true north, or magnetic bearing, using magnetic north. See also bearing and relative bearing.
    Accommodation ladder:  A portable flight of steps down a ship's side.
    Accommodation ship (or accommodation hulk):  A ship or hulk used as housing, generally when there is a lack of quarters available ashore. An operational ship can be used, but more commonly a hulk modified for accommodation is used.
    Act of Pardon or Act of Grace:  A letter from a state or power authorising action by a privateer. See also Letter of marque.
    Action Stations:  See Battle stations.
    Admiral:  Senior naval officer of Flag rank. In ascending order of seniority, Rear Admiral, Vice Admiral, Admiral and (until about 2001 when all UK five-star ranks were discontinued) Admiral of the Fleet (Royal Navy). Derivation Arabic, from Amir al-Bahr ("Ruler of the sea").
    Admiralty:  1.  A high naval authority in charge of a state's Navy or a major territorial component. In the Royal Navy (UK) the Board of Admiralty, executing the office of the Lord High Admiral, promulgates Naval law in the form of Queen's (or King's) Regulations and Admiralty Instructions. 2.  Admiralty law
    Admiralty law:  Body of law that deals with maritime cases. In the UK administered by the Probate, Divorce and Admiralty Division of the High Court of Justice or supreme court.
    Adrift:  1.  Afloat and unattached in any way to the shore or seabed, but not under way. When referring to a vessel, it implies that the vessel is not under control and therefore goes where the wind and current take her (loose from moorings or out of place). 2.  Any gear not fastened down or put away properly. 3.  Any person or thing that is misplaced or missing. When applied to a member of the navy or marine corps, such a person is "absent without leave" (AWOL) or, in US Navy and US Marine Corps terminology, is guilty of an "unauthorized absence" (UA).[8]