python - building scrapy spiders into my own program, i don't want to call scrapy from command line) -

-

in similar vein question: stackoverflow: running-multiple-spiders-in-scrapy

i wondering, can run entire scrapy project within python program? lets wanted build entire program required scraping several different sites, , build entire scrapy projects each site.

instead of running command line 1 of, want run these spiders , acquire information them.

i can use mongodb in python ok, , can build scrapy projects contain spiders, merging 1 application.

i want run application once, , have ability control multiple spiders own program

why this? application may connect other sites using api , needs compare results api site scraped site in real time. don't want ever have call scrapy command line, meant self contained.

(i have been asking lots of questions scraping recently, because trying find right solution build in)

thanks :)

yep, of course can ;)

the idea (inspired blog post) create worker , use in own python script:

from scrapy import project, signals scrapy.conf import settings scrapy.crawler import crawlerprocess scrapy.xlib.pydispatch import dispatcher multiprocessing.queues import queue import multiprocessing  class crawlerworker(multiprocessing.process):      def __init__(self, spider, result_queue):         multiprocessing.process.__init__(self)         self.result_queue = result_queue          self.crawler = crawlerprocess(settings)         if not hasattr(project, 'crawler'):             self.crawler.install()         self.crawler.configure()          self.items = []         self.spider = spider         dispatcher.connect(self._item_passed, signals.item_passed)      def _item_passed(self, item):         self.items.append(item)      def run(self):         self.crawler.crawl(self.spider)         self.crawler.start()         self.crawler.stop()         self.result_queue.put(self.items)

example of use:

result_queue = queue() crawler = crawlerworker(myspider(myargs), result_queue) crawler.start() item in result_queue.get():     yield item

another way execute scrapy crawl command system()


Comments

Post a Comment

Popular posts from this blog

c# - SVN Error : "svnadmin: E205000: Too many arguments" -

-
i trying reposotries using c# code process svncommand = null; var psi = new processstartinfo("svnadmin"); psi.redirectstandardoutput = true; psi.redirectstandarderror = true; psi.useshellexecute = false; psi.arguments= @"dump c:\repositories\myrepo > c:\temp\myrepodumpfile.dump"; using (svncommand = process.start(psi)) { var myoutput = svncommand.standardoutput; var myerror = svncommand.standarderror; debug.write("output :" + environment.newline + environment.newline + myoutput.readtoend() + environment.newline + "error :" + environment.newline + environment.newline + myerror.readtoend()); svncommand.close(); } when use dump command commandline svnadmin dump c:\repositories\myrepo > c:\temp\myrepodumpfile.dump it works fine when try ...
Read more

c# - Copy ObservableCollection to another ObservableCollection -

-
how copy observablecollection item observablecollection without reference of first collection? here observablecollection item value changes affecting both collections. code private observablecollection<ratemodel> _allmetalrate = new observablecollection<ratemodel>(); private observablecollection<ratemodel> _metalrateondate = new observablecollection<ratemodel>(); public observablecollection<ratemodel> allmetalrate { { return this._allmetalrate; } set { this._allmetalrate = value; notifypropertychanged("metalrate"); } } public observablecollection<ratemodel> metalrateondate { { return this._metalrateondate; } set { this._metalrateondate = value; notifypropertychanged("metalrateondate"); } } foreach (var item in metalrateondate) allmetalrate.add(item); what causing , how can solve it? you need clone object referenced item before it's a...
Read more

All overlapping substrings matching a java regex -

-
is there api method returns (possibly overlapping) substrings match regular expression? for example, have text string: string t = 04/31 412-555-1235; , , have pattern: pattern p = new pattern("\\d\\d+"); matches strings of 2 or more characters. the matches are: 04, 31, 412, 555, 1235. how overlapping matches? i want code return: 04, 31, 41, 412, 12, 55, 555, 55, 12, 123, 1235, 23, 235, 35. theoretically should possible -- there obvious o(n^2) algorithm enumerates , checks substrings against pattern. edit rather enumerating substrings, safer use region(int start, int end) method in matcher . checking pattern against separate, extracted substring might change result of match (e.g. if there non-capturing group or word boundary check @ start/end of pattern). edit 2 actually, it's unclear whether region() expect zero-width matches. specification vague, , experiments yield disappointing results. for example: string line = "xx90xx"; strin...
Read more