问题描述
数据来自数据库，因此如何使用此数据解析HTML，在我的代码中，我使用Cookie连接到直接指向目标页面，然后连接到set搜索表单，因此如何解析搜索。它来自后端。下面的代码首先包含我的服务连接代码，然后是搜索表单响应给出的HTML，最后是提交API post Method的搜索表单后返回到浏览器网络的html。
Connection.Response redirectResponse = Jsoup.connect(internalConstant.getSearchResultURL())
.userAgent(USER_AGENT)
.sslSocketFactory(utilService.socketFactory())
.cookies(coky)
.method(Method.GET)
.execute();
//search file
String yestarday = utilService.getYesterdayDateString();
logger.info("yestarday date: " + yestarday);
Document responseDocument = redirectResponse.parse();
FormElement searchForm = (FormElement) responseDocument.select("form[id=searchdisputesForm]").first();
checkElement("form element",searchForm);
FormElement form = (FormElement) searchForm;
Element searchField = form.select("input[name=DateFrom]").first();
checkElement("date from: ",searchField);
searchField.val(yestarday);
Element searchField1 = form.select("input[name=Dateto]").first();
checkElement("Date to: ",searchField1);
searchField1.val(yestarday);
Connection.Response searchResponse = form.submit()
.cookies(coky)
.sslSocketFactory(utilService.socketFactory())
.userAgent(USER_AGENT)
.method(Method.POST)
.ignoreHttpErrors(true)
.execute();
//and here is what gives me:
2020-10-15 11:37:38.193 INFO 3360 --- [http-nio-8095-exec-1] c.d.e.serviceImpl.UploaderTemplateImpl : searchResponse :<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<Meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>404 - File or directory not found.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;}
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;}
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS",Verdana,sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header">
<h1>Server Error</h1>
</div>
<div id="content">
<div class="content-container">
<fieldset>
<h2>404 - File or directory not found.</h2>
<h3>The resource you are looking for might have been removed,had its name changed,or is temporarily unavailable.</h3>
</fieldset>
</div>
</div>
</body>
</html>
//but the response in the network shows what I want:
<form action="/123Mobile/Portal/dispute/Searchdisputes" class="form-horizontal" enctype="multipart/form-data" id="searchdisputesForm" method="post"><input name="__RequestVerificationToken" type="hidden" value="ClhG-V26p7Aj3vi6W8tarCwHk_enagGT4mDVv2AZsU4MHATOBTbEQHuFmylooB5qvxF29aF1-wSvZZ26ijlcZJn5p_OSshL3KeqcEbAaYg01" />
<div class='col-md-12'>
<div class='panel panel-grey equal-height-column' id=''>
<div class='panel-heading'>
<h3 class='panel-title'>
<i class='fa fa-tasks'></i>
Search criteria
</h3>
</div>
<div class='panel-body'>
<div class='form-group'> <label class="col-md-2 control-label" for="DateFrom">Date From</label>
<div class='col-md-4 col-md-offset-0' id=''>
<input class="form-control datepicker" data-val="true" data-val-date="The field Date From must be a date." id="DateFrom" name="DateFrom" placeholder="" type="text" value="2020-10-14" />
<span class="field-validation-valid text-danger" data-valmsg-for="DateFrom" data-valmsg-replace="true"></span>
</div><label class="col-md-2 control-label" for="Dateto">Date To</label>
<div class='col-md-4 col-md-offset-0' id=''>
<input class="form-control datepicker" data-val="true" data-val-date="The field Date To must be a date." id="Dateto" name="Dateto" placeholder="" type="text" value="2020-10-14" />
<span class="field-validation-valid text-danger" data-valmsg-for="Dateto" data-valmsg-replace="true"></span>
</div></div>
<div class='col-md-12'>
<div class=' col-md-6'>
</div>
<button type='button' class='btn btn-success col-md-2 col-md-offset-1 ' id='ResetBtn' name='ResetBtn' onclick=''>
<span class='glyphicon glyphicon-refresh' aria-hidden='true'></span> Reset
</button>
<button type='submit' class='btn btn-primary col-md-2 col-md-offset-1 ' id='SearchBtn' name='SearchBtn' formaction='/123Mobile/Portal/Reports/SearchTransactionReport' onclick=''>
<span class='glyphicon glyphicon-search' aria-hidden='true'></span> Search
</button></div>
</div>
</div>
</div>
<div class='col-md-12'>
<div class='panel panel-grey equal-height-column' id=''>
<div class='panel-heading'>
<h3 class='panel-title'>
<i class='fa fa-tasks'></i>
Search results
</h3>
</div>
<div class='panel-body'>
<div style='' class='table-responsive'><table class='table table-hover table-striped'><thead><tr><th style='width:15%;'>File Name</th><th style='width:10%;'>Scheme</th><th style='width:15%;'>Proccessing Date</th><th style='width:8%;'></th><th style='width:8%;'></th></tr> </thead><tbody><tr><td style='-moz-word-break: break-all; -o-word-break: break-all; word-break: break-word;-ms-word-break: break-all;overflow-wrap: break-word;word-wrap: break-word; width: 15%;' class=''>Transaction-FIBWallet-20201014</td><td style='-moz-word-break: break-all; -o-word-break: break-all; word-break: break-word;-ms-word-break: break-all;overflow-wrap: break-word;word-wrap: break-word; width: 10%;' class=''>FIBWallet</td><td style='-moz-word-break: break-all; -o-word-break: break-all; word-break: break-word;-ms-word-break: break-all;overflow-wrap: break-word;word-wrap: break-word; width: 15%;' class=''>2020-10-14</td><td style='width: 8%;' class=''><a href="/123Mobile/Portal/Reports/DownloadTransactionReport/40072">Download as Text </a></td><td style='width: 8%;' class=''><a href="/123Mobile/Portal/Reports/DownloadTransactionReport/40073">Download as Excel </a></td></tr></tbody></table></div>
</div>
</div>
</div>
<div class='col-md-12'> <div class="pagination-container"><ul class="pagination"><li class="disabled PagedList-skipToFirst"><a>««</a></li><li class="disabled PagedList-skipToPrevIoUs"><a rel="prev">«</a></li><li class="active"><a>1</a></li><li class="disabled PagedList-skipToNext"><a rel="next">»</a></li><li class="disabled PagedList-skipToLast"><a>»»</a></li></ul></div></div>
</form>
解决方法
除非您需要从服务器获取带有标题的会话cookie和其他信息，否则我将避免以GET请求开头并直接提交POST表单请求。
您说得到的回复看起来像
>>> np.where(Q5[0]>Q5[0,0],Q5[0],-1)[1]
2
这表明您在发出POST请求时未跨页面发送正确的标头，或者该页面是一个单页面应用程序，它会发出一些后续（异步）请求，然后修改页面源。
除了说在网络选项卡上使用Chrome开发工具来调查POST请求发生时向服务器发送了哪些请求外，我对所提供的信息真的无能为力。看标题。
典型的标头涉及某种会话cookie，可能是引荐来源标头。但是，如果这两个单独的方法无法满足您的要求，则需要确切地开始复制请求...可能需要以您需要的形式获得CSRF令牌（在这种情况下，初始的GET请求对于获得一个有效的令牌以发送回POST）。