Tag: 机械化

如何使用Mechanize gem获取网站的所有链接？: 如何使用ruby Mechanize gem获取网站的所有链接？ Mechanize可以像Anemone gem那样做： Anemone.crawl(“https://www.google.com.vn/”) do |anemone| anemone.on_every_page do |page| puts page.url end end 我是网络爬虫的新手。提前致谢！

Rails 3 Mechanize – SocketError：getaddrinfo：主机或名称未知: 我正在使用机械化，我得到这个错误。有人可以帮帮我吗。我已将元刷新设置为true。错误日志： SocketError: getaddrinfo: Host or name not known form C:/Ruby192/lib/ruby/1.9.1/net/http.rb:644:in ‘initiallize’ form C:/Ruby192/lib/ruby/1.9.1/net/http.rb:644:in ‘open’ form C:/Ruby192/lib/ruby/1.9.1/net/http.rb:644:in ‘block in connect’ form C:/Ruby192/lib/ruby/1.9.1/timeout.rb:44:in ‘timeout’ form C:/Ruby192/lib/ruby/1.9.1/timeout.rb:87:in ‘timeout’ form C:/Ruby192/lib/ruby/1.9.1/net/http.rb:644:in ‘connect’ form C:/Ruby192/lib/ruby/1.9.1/net/http.rb:637:in ‘do_start’ form C:/Ruby192/lib/ruby/1.9.1/net/http.rb:632:in ‘start’ form C:/Ruby192/lib/ruby/gems/1.9.1/gems/mechanize-1.0.0/lib/mechanize.rb:527:in ‘fetch_page’ form C:/Ruby192/lib/ruby/gems/1.9.1/gems/mechanize-1.0.0/lib/mechanize.rb:591:in ‘fetch_page’ form C:/Ruby192/lib/ruby/gems/1.9.1/gems/mechanize-1.0.0/lib/mechanize.rb:464:in ‘post_form’ form C:/Ruby192/lib/ruby/gems/1.9.1/gems/mechanize-1.0.0/lib/mechanize.rb:370:in ‘submit’ form C:/Ruby192/lib/ruby/gems/1.9.1/gems/mechanize-1.0.0/lib/form.rb:141:in ‘submit’ 更新 – 表单提交http://sofzh.miximages.com/ruby-on-rails/Cuo9u.png上的错误消息 […]

Mechanize – 如何在rails中关注或“单击”Meta刷新: 我对Mechanize有点麻烦。当使用Mechanize提交表单时。我来一个页面有一个元刷新，没有链接。我的问题是我如何遵循元刷新？我试图允许元刷新，但后来我得到一个套接字错误。示例代码 require ‘mechanize’ agent = WWW::Mechanize.new agent.get(“http://euroads.dk”) form = agent.page.forms.first form.username = “username” form.password = “password” form.submit page = agent.get(“http://www.euroads.dk/system/index.php?showpage=login”) agent.page.body 响应：然后我尝试： redirect_url = page.parser.at(‘META[HTTP-EQUIV=\”Refresh\”]’)[ “0;URL=index.php?showpage=m_frontpage\”][/url=(.+)/, 1] 但我得到： NoMethodError：nil的未定义方法'[]’：NilClass

获得机械化来浏览x个链接并获得所有标题？: 基本上我想使用mechanize来浏览本网站上az的所有页面http://www.tv.com/shows/sort/a_z/ 然后，对于每个字母，在字母“a”的所有页面上获得每个节目的标题。目前我正试图让它与字母“a”一起工作。这是我到目前为止，但不知道从哪里去？ require ‘mechanize’ agent=Mechanize.new goog = agent.get “http://www.tv.com/shows/sort/a_z/” search = goog.link_with(:href => “/shows/sort/a/”).click

如果只有一个字段为粗体，如何解析一行？ Nokogiri和Ruby: 所以我有这个代码收集我需要的所有产品信息： # get main page page = agent.get “http://www.site.com.mx/tienda/index.php” search_form = page.forms.first search_result = agent.submit search_form doc = Nokogiri::HTML(search_result.body) rows = doc.css(“table.articulos tr”) i = 0 details = rows.collect do |row| detail = {} [ [:sku, ‘td[3]/text()’], [:desc, ‘td[4]/text()’], [:qty, ‘td[5]/text()’], [:qty2, ‘td[5]/p/b/text()’], [:price, ‘td[6]/text()’] ].collect do |name, xpath| detail[name] = row.at_xpath(xpath).to_s.strip end i = […]

在Windows 7上使用Ruby 2.3安装机械化时出错: 我正在尝试在Windows 7上安装带有Ruby 2.3的Mechanize。但是我收到了以下错误。有人能指出我正确的方向吗？ PS C:\DevKit> ruby –version ruby 2.3.0p0 (2015-12-25 revision 53290) [x64-mingw32] PS C:\DevKit> gem install mechanize Fetching: net-http-digest_auth-1.4.gem (100%) Successfully installed net-http-digest_auth-1.4 Fetching: net-http-persistent-2.9.4.gem (100%) Successfully installed net-http-persistent-2.9.4 Fetching: mime-types-2.99.1.gem (100%) Successfully installed mime-types-2.99.1 Fetching: unf_ext-0.0.7.2-x64-mingw32.gem (100%) Successfully installed unf_ext-0.0.7.2-x64-mingw32 Fetching: unf-0.1.4.gem (100%) Successfully installed unf-0.1.4 Fetching: domain_name-0.5.20160310.gem (100%) Successfully installed domain_name-0.5.20160310 […]

使用Ruby Mechanize来擦除所有连续页面: 我正在寻找帮助，以最好的方式循环网站上的连续页面，同时从每个页面抓取相关数据。例如，我想去一个特定的网站（以下例子中的craigslist），从第一页抓取数据，转到下一页，抓取所有相关数据等，直到最后一页。在我的脚本中，我使用了一个while循环，因为它似乎对我来说最有意义。但是，它似乎没有正常工作，只是从第一页抓取数据。熟悉Ruby / Mechanize的人可以指出我正确的方向，完成这项任务的最佳方法是什么。我花了无数个小时试图解决这个问题，觉得我错过了一些非常基本的东西。在此先感谢您的帮助。 require ‘mechanize’ require ‘pry’ # initialze agent = Mechanize.new { |agent| agent.user_agent_alias = ‘Mac Safari’} url = “http://charlotte.craigslist.org/search/rea” page = agent.get(url) # Create an empty array to dump contents into property_results = [] # Scrape all successive pages from craigslist while page.link_with(:dom_class => “button next”) != […]

Ruby Mechanize屏幕抓取帮助: 我试图在一张桌子上刮一排日期。我想只刮掉今天有第三行的日期。这是我的机械化代码。我正在尝试选择colum row witch今天的日期及其及其列： agent.page.search（ “// TD”）的地图（：文本）。.MAP（＆：条） Output: “11-02-2011”, “1”, “1”, “1”, “1”, “0”, “0,00 DKK”, “0,00”, “0,00 DKK”, “12-02-2011”, “5”, “5”, “1”, “4”, “0”, “0,00 DKK”, “0,00”, “0,00 DKK”, “14-02-2011”, “1”, “3”, “1”, “1”, “0”, “0,00 DKK”, “,00”, “0,00 DKK”, “7”, “9”, “3”, “6”, “0”, “0,00 DKK”, “0,00”, “0,00 DKK “ 我想只刮掉今天的第三行。

用机械化单击图像: 单击文本agent.click(page.link_with(:text => ‘some_text’)与mechainze是小菜一碟。如何用机械化点击图像？

机械化Ruby 1.9.3编码问题: 使用以下代码（来自Mechanize站点，但在rake任务中）.. namespace :ans do task :grab => :environment do a = Mechanize.new { |agent| agent.user_agent_alias = ‘Mac Safari’ } begin a.get(‘http://google.com/’) do |page| search_result = page.form_with(:name => ‘f’) do |search| search.q = ‘Hello world’ end.submit search_result.links.each do |link| puts link.text end end end end end 我收到编码错误.. rake aborted! “\x8B” from ASCII-8BIT to UTF-8 这是在使用以下内容时： Rails […]