问题描述
我的公司使用C＃.net应用程序以编程方式从供应商网站上以CSV文件的形式检索数据。但是，供应商最近将其API更改为JSON API。我是一个新手程序员，对JSON API的调用没有任何经验。当供应商注册我的新API用户ID时，他们提供了如何使用 curl 下载必要的csv数据文件的示例。这是示例：
echo 'Going to log in to sso'
data=$(curl -c cookies.txt -X POST https://sso.theice.com/api/authenticateTfa -H "Content-Type: application/json" \
--data '{"userId":"yourApiUserId","password":"yourpassword","appKey":"yourAppKey"}')
echo 'Going to download icecleared_oil_2020_06_26.dat file from Settlement_Reports_CSV/Oil'
curl -b cookies.txt -O ' https://downloads2.theice.com/Settlement_Reports_CSV/Oil/icecleared_oil_2020_06_26.dat
由于我没有Curl的经验，因此我使用https://curl.olsh.me/将上面的示例转换为C＃。这样做会生成以下代码：
//Authenticate userId (and get cookie?)
using (var httpClient = new HttpClient())
{
using (var request = new HttpRequestMessage(new HttpMethod("POST"),"https://sso.theice.com/api/authenticateTfa"))
{
request.Content = new StringContent("{\"userId\":\"yourApiUserId\",\"password\":\"yourpassword\",\"appKey\":\"yourAppKey\"}");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
var response = await httpClient.SendAsync(request);
}
}
//Using authentication cookie,from prevIoUs code block,retrieve dat file
var handler = new httpclienthandler();
handler.UseCookies = false;
using (var httpClient = new HttpClient(handler))
{
using (var request = new HttpRequestMessage(new HttpMethod("GET"),"https://downloads2.theice.com/Settlement_Reports_CSV/Oil/icecleared_oil_2020_06_26.dat"))
{
request.Headers.TryAddWithoutValidation("Cookie","cookies.txt");
var response = await httpClient.SendAsync(request);
}
}
我将此代码集成到我的项目中，并能够获取它来验证我的userID（POST请求），但是我一生都无法弄清楚如何从第二个代码块（GET请求）中检索DAT文件）。我观察到，无论我是否成功登录POST请求，GET请求的响应都是相同的。此时，我假设GET请求没有从POST请求中获取验证cookie，但是即使这样，我也不知道如何从GET请求中的响应对象中提取文件。>
有人可以建议我需要做什么吗？下面是已集成到我的项目中的代码：
public async Task DownloadAsync(List<string> iceReportFileNamesFullUrl)
{
await AuthenticateUserCredentialsOnICE();
await RetrieveIceRportFiles(iceReportFileNamesFullUrl);
}
private async Task AuthenticateUserCredentialsOnICE()
{
using (HttpClient httpClient = new HttpClient())
{
using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("POST"),"https://sso.theice.com/api/authenticateTfa"))
{
request.Content = new StringContent("{\"userId\":\"xxxxxxxxx\",\"password\":\"xxxxxxxxxx\",\"appKey\":\"xxxxxxxxxxx\"}");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
HttpResponseMessage response = await httpClient.SendAsync(request);
var value = response.Content.ReadAsstringAsync();
}
}
}
private static async Task RetrieveIceRportFiles(List<string> iceReportFileNamesFullUrl)
{
foreach (string iceReportUrl in iceReportFileNamesFullUrl)
{
httpclienthandler handler = new httpclienthandler();
handler.UseCookies = false;
using (HttpClient httpClient = new HttpClient(handler))
{
using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("GET"),iceReportUrl))
{
request.Headers.TryAddWithoutValidation("Cookie","cookies.txt");
HttpResponseMessage response = await httpClient.SendAsync(request);
var value = response.Content.ReadAsstringAsync();
}
}
}
}
-更新：2020年8月25日-
谢谢大家的评论。您清楚地表明，我没有从POST请求中捕获Cookie。经过一些研究（this，this和this），我更新了代码，以便现在可以成功下载CSV / DAT文件。我已经发布了下面的内容，尽管它仍然需要错误处理，登录失败的检查等。我也肯定会有关键的反馈。
这里是：
private CookieContainer authenticationCookieContainer;
//main method for downloading file
public async Task DownloadAsync(string iceReportDownloadUrl)
{
await AuthenticateUserCredentialsOnIceAsync();
await SendDownloadReportRequestAsync(iceReportDownloadUrl);
}
通过JSON API对凭据进行身份验证，并将身份验证cookie存储在authenticationCookieContainter中
private async Task AuthenticateUserCredentialsOnIceAsync()
{
//This will contain the ICE authentication cookies
//if the login is successful
authenticationCookieContainer = new CookieContainer();
//Create and assign handler for proxy config and cookie storage
using (httpclienthandler handler = new httpclienthandler() { CookieContainer = authenticationCookieContainer })
{
using (HttpClient httpClient = new HttpClient(handler))
{
using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("POST"),"https://sso.theice.com/api/authenticateTfa"))
{
request.Content = new StringContent("{\"userId\":\"xxxxxxxxxxx\",\"password\":\"xxxxxxxxxxxxxx\",\"appKey\":\"xxxxxxxxxxxxx\"}");
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
HttpResponseMessage response = await httpClient.SendAsync(request);
}
}
}
}
如果登录成功，则authenticationCookieContainer将包含 登录cookie，并允许您在指定的位置请求数据 网址
private async Task SendDownloadReportRequestAsync(string iceReportDownloadUrl)
{
if (authenticationCookieContainer != null)
{
//Create and assign handler for proxy config and cookie storage
using (httpclienthandler handler = new httpclienthandler() { CookieContainer = authenticationCookieContainer })
{
//Set to true to use the cookies obtained during authentication call
handler.UseCookies = true;
using (HttpClient httpClient = new HttpClient(handler))
{
Uri iceReportUri = new Uri(iceReportDownloadUrl);
//Request the ICE data file using the url passed through
using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("GET"),iceReportUri.AbsoluteUri))
{
HttpResponseMessage response = await httpClient.SendAsync(request);
using (Stream responseStream = await response.Content.ReadAsstreamAsync())
{
//Write the extracted file to the file path
byte[] content = StreamToByteArray(responseStream);
File.WriteallBytes(@"C:\Users\my.name\Desktop\" + iceReportUri.Segments.Last(),content);
}
}
}
}
}
}
private static byte[] StreamToByteArray(Stream stream)
{
using (MemoryStream memoryStream = new MemoryStream())
{
stream.copyTo(memoryStream);
return memoryStream.ToArray();
}
}
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）