download web Dir


Author: Dave
Date: 07.07.17 - 10:55am



quick python script to download all files of a certain extension from a html web directory to a local folder.

make sure you have beautiful soup installed: pip install beautifulsoup4

todo: create folder if it doesnt exist

from urllib2 import urlopen
from bs4 import BeautifulSoup

webDir = 'https://indy.fulgan.com/SSL/'
localDir = 'c:\\webdl2\\'

urlpath =urlopen(webDir)
html = urlpath.read().decode('utf-8')

soup = BeautifulSoup(html,"html.parser")
links = soup.find_all('a')

filelist = []

for tag in links:
    link = tag.get('href',None)
    if link is not None and link.find('.zip') != -1:
        #print link
        filelist.append(link)

print('%d files to dl...' % len(filelist) )

for i, filename in enumerate(filelist):
    remotefile = urlopen(webDir + filename)
    localfile = open(localDir + filename,'wb')
    localfile.write(remotefile.read())
    localfile.close()
    remotefile.close()
    print('%d/%d' % (i , len(filelist)))

print('done')





Comments: (0)

 
Leave Comment:
Name:
Email: (not shown)
Message: (Required)
Math Question: 75 + 89 = ? followed by the letter: Z 



About Me
More Blogs
Main Site
Posts: (All)
2020 ( 2 )
2019 ( 5 )
2018 ( 6 )
2017 (6)
     Python and VB6
     Python pros and cons
     download web Dir
     vc rand in python
     VB6 Language Enhancement
     Register .NET as COM
2016 ( 22 )
2015 ( 16 )
2014 ( 25 )
2013 ( 4 )
2012 ( 10 )
2011 ( 7 )
2010 ( 11 )
2009 ( 3 )