1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. Note: prior to version 11, this would be embedded_httpd_processes on Linux and embedded_httpd_threads everywhere else. It now means embedded_httpd_hybrid everywhere supported and embedded_httpd_threads everywhere else. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 490 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 491 492 /++ 493 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 494 +/ 495 version(Demo) 496 unittest { 497 import arsd.cgi; 498 499 mixin DispatcherMain!( 500 "/".serveStaticFileDirectory(null, true) 501 ); 502 } 503 504 /++ 505 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 506 +/ 507 version(Demo) 508 unittest { 509 import arsd.cgi; 510 511 void requestHandler(Cgi cgi) { 512 cgi.dispatcher!( 513 "/".serveStaticFileDirectory(null, true) 514 ); 515 } 516 517 // mixin GenericMain!requestHandler would add this function: 518 void main(string[] args) { 519 // this is all the content of [cgiMainImpl] which you can also call 520 521 // cgi.d embeds a few add on functions like real time event forwarders 522 // and session servers it can run in other processes. this spawns them, if needed. 523 if(tryAddonServers(args)) 524 return; 525 526 // cgi.d allows you to easily simulate http requests from the command line, 527 // without actually starting a server. this function will do that. 528 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 529 return; 530 531 RequestServer server; 532 // you can change the default port here if you like 533 // server.listeningPort = 9000; 534 535 // then call this to let the command line args override your default 536 server.configureFromCommandLine(args); 537 538 // here is where you could print out the listeningPort to the user if you wanted 539 540 // and serve the request(s) according to the compile configuration 541 server.serve!(requestHandler)(); 542 543 // or you could explicitly choose a serve mode like this: 544 // server.serveEmbeddedHttp!requestHandler(); 545 } 546 } 547 548 /++ 549 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 550 otherwise run through the rest of the internal mechanisms to call your functions without actually 551 spinning up a server. 552 +/ 553 version(Demo) 554 unittest { 555 import arsd.cgi; 556 557 void requestHandler(Cgi cgi) { 558 559 } 560 561 // D doesn't let me embed a unittest inside an example unittest 562 // so this is a function, but you can do it however in your real program 563 /* unittest */ void runTests() { 564 auto tester = new CgiTester(&requestHandler); 565 566 auto response = tester.GET("/"); 567 assert(response.code == 200); 568 } 569 } 570 571 /++ 572 The session system works via a built-in spawnable server. 573 574 Bugs: 575 Requires addon servers, which are not implemented yet on Windows. 576 +/ 577 version(Posix) 578 version(Demo) 579 unittest { 580 import arsd.cgi; 581 582 struct SessionData { 583 string userId; 584 } 585 586 void handler(Cgi cgi) { 587 auto session = cgi.getSessionObject!SessionData; 588 589 if(cgi.pathInfo == "/login") { 590 session.userId = cgi.queryString; 591 cgi.setResponseLocation("view"); 592 } else { 593 cgi.write(session.userId); 594 } 595 } 596 597 mixin GenericMain!handler; 598 } 599 600 static import std.file; 601 602 static import arsd.core; 603 version(Posix) 604 import arsd.core : makeNonBlocking; 605 606 607 // for a single thread, linear request thing, use: 608 // -version=embedded_httpd_threads -version=cgi_no_threads 609 610 version(Posix) { 611 version(CRuntime_Musl) { 612 613 } else version(minimal) { 614 615 } else { 616 version(FreeBSD) { 617 // I never implemented the fancy stuff there either 618 } else { 619 version=with_breaking_cgi_features; 620 version=with_sendfd; 621 version=with_addon_servers; 622 } 623 } 624 } 625 626 version(Windows) { 627 version(minimal) { 628 629 } else { 630 // not too concerned about gdc here since the mingw version is fairly new as well 631 version=with_breaking_cgi_features; 632 } 633 } 634 635 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 636 void cloexec(int fd) { 637 version(Posix) { 638 import core.sys.posix.fcntl; 639 fcntl(fd, F_SETFD, FD_CLOEXEC); 640 } 641 } 642 643 void cloexec(Socket s) { 644 version(Posix) { 645 import core.sys.posix.fcntl; 646 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 647 } 648 } 649 650 // the servers must know about the connections to talk to them; the interfaces are vital 651 version(with_addon_servers) 652 version=with_addon_servers_connections; 653 654 version(embedded_httpd) { 655 version(OSX) 656 version = embedded_httpd_threads; 657 else 658 version=embedded_httpd_hybrid; 659 /* 660 version(with_openssl) { 661 pragma(lib, "crypto"); 662 pragma(lib, "ssl"); 663 } 664 */ 665 } 666 667 version(embedded_httpd_hybrid) { 668 version=embedded_httpd_threads; 669 version(cgi_no_fork) {} else version(Posix) 670 version=cgi_use_fork; 671 version=cgi_use_fiber; 672 } 673 674 version(cgi_use_fork) 675 enum cgi_use_fork_default = true; 676 else 677 enum cgi_use_fork_default = false; 678 679 version(embedded_httpd_processes) 680 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 681 682 version(embedded_httpd_threads) { 683 // unless the user overrides the default.. 684 version(cgi_session_server_process) 685 {} 686 else 687 version=cgi_embedded_sessions; 688 } 689 version(scgi) { 690 // unless the user overrides the default.. 691 version(cgi_session_server_process) 692 {} 693 else 694 version=cgi_embedded_sessions; 695 } 696 697 // fall back if the other is not defined so we can cleanly version it below 698 version(cgi_embedded_sessions) {} 699 else version=cgi_session_server_process; 700 701 702 version=cgi_with_websocket; 703 704 enum long defaultMaxContentLength = 5_000_000; 705 706 /* 707 708 To do a file download offer in the browser: 709 710 cgi.setResponseContentType("text/csv"); 711 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 712 */ 713 714 // FIXME: the location header is supposed to be an absolute url I guess. 715 716 // FIXME: would be cool to flush part of a dom document before complete 717 // somehow in here and dom.d. 718 719 720 // these are public so you can mixin GenericMain. 721 // FIXME: use a function level import instead! 722 public import std.string; 723 public import std.stdio; 724 public import std.conv; 725 import std.uri; 726 import std.uni; 727 import std.algorithm.comparison; 728 import std.algorithm.searching; 729 import std.exception; 730 import std.base64; 731 static import std.algorithm; 732 import std.datetime; 733 import std.range; 734 735 import std.process; 736 737 import std.zlib; 738 739 740 T[] consume(T)(T[] range, int count) { 741 if(count > range.length) 742 count = range.length; 743 return range[count..$]; 744 } 745 746 int locationOf(T)(T[] data, string item) { 747 const(ubyte[]) d = cast(const(ubyte[])) data; 748 const(ubyte[]) i = cast(const(ubyte[])) item; 749 750 // this is a vague sanity check to ensure we aren't getting insanely 751 // sized input that will infinite loop below. it should never happen; 752 // even huge file uploads ought to come in smaller individual pieces. 753 if(d.length > (int.max/2)) 754 throw new Exception("excessive block of input"); 755 756 for(int a = 0; a < d.length; a++) { 757 if(a + i.length > d.length) 758 return -1; 759 if(d[a..a+i.length] == i) 760 return a; 761 } 762 763 return -1; 764 } 765 766 /// If you are doing a custom cgi class, mixing this in can take care of 767 /// the required constructors for you 768 mixin template ForwardCgiConstructors() { 769 this(long maxContentLength = defaultMaxContentLength, 770 string[string] env = null, 771 const(ubyte)[] delegate() readdata = null, 772 void delegate(const(ubyte)[]) _rawDataOutput = null, 773 void delegate() _flush = null 774 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 775 776 this(string[] args) { super(args); } 777 778 this( 779 BufferedInputRange inputData, 780 string address, ushort _port, 781 int pathInfoStarts = 0, 782 bool _https = false, 783 void delegate(const(ubyte)[]) _rawDataOutput = null, 784 void delegate() _flush = null, 785 // this pointer tells if the connection is supposed to be closed after we handle this 786 bool* closeConnection = null) 787 { 788 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 789 } 790 791 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 792 } 793 794 /// thrown when a connection is closed remotely while we waiting on data from it 795 class ConnectionClosedException : Exception { 796 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 797 super(message, file, line, next); 798 } 799 } 800 801 802 version(Windows) { 803 // FIXME: ugly hack to solve stdin exception problems on Windows: 804 // reading stdin results in StdioException (Bad file descriptor) 805 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 806 private struct stdin { 807 struct ByChunk { // Replicates std.stdio.ByChunk 808 private: 809 ubyte[] chunk_; 810 public: 811 this(size_t size) 812 in { 813 assert(size, "size must be larger than 0"); 814 } 815 do { 816 chunk_ = new ubyte[](size); 817 popFront(); 818 } 819 820 @property bool empty() const { 821 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 822 } 823 @property nothrow ubyte[] front() { return chunk_; } 824 void popFront() { 825 enforce(!empty, "Cannot call popFront on empty range"); 826 chunk_ = stdin.rawRead(chunk_); 827 } 828 } 829 830 import core.sys.windows.windows; 831 static: 832 833 T[] rawRead(T)(T[] buf) { 834 uint bytesRead; 835 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 836 837 if (!result) { 838 auto err = GetLastError(); 839 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 840 return buf[0..0]; 841 // Some other error, throw it 842 843 char* buffer; 844 scope(exit) LocalFree(buffer); 845 846 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 847 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 848 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 849 throw new Exception(to!string(buffer)); 850 } 851 enforce(!(bytesRead % T.sizeof), "I/O error"); 852 return buf[0..bytesRead / T.sizeof]; 853 } 854 855 auto byChunk(size_t sz) { return ByChunk(sz); } 856 857 void close() { 858 std.stdio.stdin.close; 859 } 860 } 861 } 862 863 /// The main interface with the web request 864 class Cgi { 865 public: 866 /// the methods a request can be 867 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 868 // these are defined in the standard, but idk if they are useful for anything 869 OPTIONS, TRACE, CONNECT, 870 // These seem new, I have only recently seen them 871 PATCH, MERGE, 872 // this is an extension for when the method is not specified and you want to assume 873 CommandLine } 874 875 876 /+ 877 878 ubyte[] perRequestMemoryPool; 879 void[] perRequestMemoryPoolWithPointers; 880 // might want to just slice the buffer itself too when we happened to have gotten a full request inside it and don't need to decode 881 // then the buffer also can be recycled if it is set. 882 883 // we might also be able to set memory recyclable true by default, but then the property getters set it to false. but not all the things are property getters. but realistically anything except benchmarks are gonna get something lol so meh. 884 885 /+ 886 struct VariableCollection { 887 string[] opIndex(string name) { 888 889 } 890 } 891 892 /++ 893 Call this to indicate that you've not retained any reference to the request-local memory (including all strings returned from the Cgi object) outside the request (you can .idup anything you need to store) and it is thus free to be freed or reused by another request. 894 895 Most handlers should be able to call this; retaining memory is the exception in any cgi program, but since I can't prove it from inside the library, it plays it safe and lets the GC manage it unless you opt into this behavior. All cgi.d functions will duplicate strings if needed (e.g. session ids from cookies) so unless you're doing something yourself, this should be ok. 896 897 History: 898 Added 899 +/ 900 public void recycleMemory() { 901 902 } 903 +/ 904 905 906 /++ 907 Cgi provides a per-request memory pool 908 909 +/ 910 void[] allocateMemory(size_t nBytes) { 911 912 } 913 914 /// ditto 915 void[] reallocateMemory(void[] old, size_t nBytes) { 916 917 } 918 919 /// ditto 920 void freeMemory(void[] memory) { 921 922 } 923 +/ 924 925 926 /* 927 import core.runtime; 928 auto args = Runtime.args(); 929 930 we can call the app a few ways: 931 932 1) set up the environment variables and call the app (manually simulating CGI) 933 2) simulate a call automatically: 934 ./app method 'uri' 935 936 for example: 937 ./app get /path?arg arg2=something 938 939 Anything on the uri is treated as query string etc 940 941 on get method, further args are appended to the query string (encoded automatically) 942 on post method, further args are done as post 943 944 945 @name means import from file "name". if name == -, it uses stdin 946 (so info=@- means set info to the value of stdin) 947 948 949 Other arguments include: 950 --cookie name=value (these are all concated together) 951 --header 'X-Something: cool' 952 --referrer 'something' 953 --port 80 954 --remote-address some.ip.address.here 955 --https yes 956 --user-agent 'something' 957 --userpass 'user:pass' 958 --authorization 'Basic base64encoded_user:pass' 959 --accept 'content' // FIXME: better example 960 --last-event-id 'something' 961 --host 'something.com' 962 --session name=value (these are added to a mock session, changes to the session are printed out as dummy response headers) 963 964 Non-simulation arguments: 965 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 966 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 967 968 */ 969 970 /** Initializes it with command line arguments (for easy testing) */ 971 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 972 rawDataOutput = _rawDataOutput; 973 // these are all set locally so the loop works 974 // without triggering errors in dmd 2.064 975 // we go ahead and set them at the end of it to the this version 976 int port; 977 string referrer; 978 string remoteAddress; 979 string userAgent; 980 string authorization; 981 string origin; 982 string accept; 983 string lastEventId; 984 bool https; 985 string host; 986 RequestMethod requestMethod; 987 string requestUri; 988 string pathInfo; 989 string queryString; 990 991 bool lookingForMethod; 992 bool lookingForUri; 993 string nextArgIs; 994 995 string _cookie; 996 string _queryString; 997 string[][string] _post; 998 string[string] _headers; 999 1000 string[] breakUp(string s) { 1001 string k, v; 1002 auto idx = s.indexOf("="); 1003 if(idx == -1) { 1004 k = s; 1005 } else { 1006 k = s[0 .. idx]; 1007 v = s[idx + 1 .. $]; 1008 } 1009 1010 return [k, v]; 1011 } 1012 1013 lookingForMethod = true; 1014 1015 scriptName = args[0]; 1016 scriptFileName = args[0]; 1017 1018 environmentVariables = cast(const) environment.toAA; 1019 1020 foreach(arg; args[1 .. $]) { 1021 if(arg.startsWith("--")) { 1022 nextArgIs = arg[2 .. $]; 1023 } else if(nextArgIs.length) { 1024 if (nextArgIs == "cookie") { 1025 auto info = breakUp(arg); 1026 if(_cookie.length) 1027 _cookie ~= "; "; 1028 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 1029 } 1030 if (nextArgIs == "session") { 1031 auto info = breakUp(arg); 1032 _commandLineSession[info[0]] = info[1]; 1033 } 1034 1035 else if (nextArgIs == "port") { 1036 port = to!int(arg); 1037 } 1038 else if (nextArgIs == "referrer") { 1039 referrer = arg; 1040 } 1041 else if (nextArgIs == "remote-address") { 1042 remoteAddress = arg; 1043 } 1044 else if (nextArgIs == "user-agent") { 1045 userAgent = arg; 1046 } 1047 else if (nextArgIs == "authorization") { 1048 authorization = arg; 1049 } 1050 else if (nextArgIs == "userpass") { 1051 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 1052 } 1053 else if (nextArgIs == "origin") { 1054 origin = arg; 1055 } 1056 else if (nextArgIs == "accept") { 1057 accept = arg; 1058 } 1059 else if (nextArgIs == "last-event-id") { 1060 lastEventId = arg; 1061 } 1062 else if (nextArgIs == "https") { 1063 if(arg == "yes") 1064 https = true; 1065 } 1066 else if (nextArgIs == "header") { 1067 string thing, other; 1068 auto idx = arg.indexOf(":"); 1069 if(idx == -1) 1070 throw new Exception("need a colon in a http header"); 1071 thing = arg[0 .. idx]; 1072 other = arg[idx + 1.. $]; 1073 _headers[thing.strip.toLower()] = other.strip; 1074 } 1075 else if (nextArgIs == "host") { 1076 host = arg; 1077 } 1078 // else 1079 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1080 1081 nextArgIs = null; 1082 } else if(lookingForMethod) { 1083 lookingForMethod = false; 1084 lookingForUri = true; 1085 1086 if(arg.asLowerCase().equal("commandline")) 1087 requestMethod = RequestMethod.CommandLine; 1088 else 1089 requestMethod = to!RequestMethod(arg.toUpper()); 1090 } else if(lookingForUri) { 1091 lookingForUri = false; 1092 1093 requestUri = arg; 1094 1095 auto idx = arg.indexOf("?"); 1096 if(idx == -1) 1097 pathInfo = arg; 1098 else { 1099 pathInfo = arg[0 .. idx]; 1100 _queryString = arg[idx + 1 .. $]; 1101 } 1102 } else { 1103 // it is an argument of some sort 1104 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1105 auto parts = breakUp(arg); 1106 _post[parts[0]] ~= parts[1]; 1107 allPostNamesInOrder ~= parts[0]; 1108 allPostValuesInOrder ~= parts[1]; 1109 } else { 1110 if(_queryString.length) 1111 _queryString ~= "&"; 1112 auto parts = breakUp(arg); 1113 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1114 } 1115 } 1116 } 1117 1118 acceptsGzip = false; 1119 keepAliveRequested = false; 1120 requestHeaders = cast(immutable) _headers; 1121 1122 cookie = _cookie; 1123 cookiesArray = getCookieArray(); 1124 cookies = keepLastOf(cookiesArray); 1125 1126 queryString = _queryString; 1127 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1128 get = keepLastOf(getArray); 1129 1130 postArray = cast(immutable) _post; 1131 post = keepLastOf(_post); 1132 1133 // FIXME 1134 filesArray = null; 1135 files = null; 1136 1137 isCalledWithCommandLineArguments = true; 1138 1139 this.port = port; 1140 this.referrer = referrer; 1141 this.remoteAddress = remoteAddress; 1142 this.userAgent = userAgent; 1143 this.authorization = authorization; 1144 this.origin = origin; 1145 this.accept = accept; 1146 this.lastEventId = lastEventId; 1147 this.https = https; 1148 this.host = host; 1149 this.requestMethod = requestMethod; 1150 this.requestUri = requestUri; 1151 this.pathInfo = pathInfo; 1152 this.queryString = queryString; 1153 this.postBody = null; 1154 } 1155 1156 private { 1157 string[] allPostNamesInOrder; 1158 string[] allPostValuesInOrder; 1159 string[] allGetNamesInOrder; 1160 string[] allGetValuesInOrder; 1161 } 1162 1163 CgiConnectionHandle getOutputFileHandle() { 1164 return _outputFileHandle; 1165 } 1166 1167 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1168 1169 /** Initializes it using a CGI or CGI-like interface */ 1170 this(long maxContentLength = defaultMaxContentLength, 1171 // use this to override the environment variable listing 1172 in string[string] env = null, 1173 // and this should return a chunk of data. return empty when done 1174 const(ubyte)[] delegate() readdata = null, 1175 // finally, use this to do custom output if needed 1176 void delegate(const(ubyte)[]) _rawDataOutput = null, 1177 // to flush teh custom output 1178 void delegate() _flush = null 1179 ) 1180 { 1181 1182 // these are all set locally so the loop works 1183 // without triggering errors in dmd 2.064 1184 // we go ahead and set them at the end of it to the this version 1185 int port; 1186 string referrer; 1187 string remoteAddress; 1188 string userAgent; 1189 string authorization; 1190 string origin; 1191 string accept; 1192 string lastEventId; 1193 bool https; 1194 string host; 1195 RequestMethod requestMethod; 1196 string requestUri; 1197 string pathInfo; 1198 string queryString; 1199 1200 1201 1202 isCalledWithCommandLineArguments = false; 1203 rawDataOutput = _rawDataOutput; 1204 flushDelegate = _flush; 1205 auto getenv = delegate string(string var) { 1206 if(env is null) 1207 return std.process.environment.get(var); 1208 auto e = var in env; 1209 if(e is null) 1210 return null; 1211 return *e; 1212 }; 1213 1214 environmentVariables = env is null ? 1215 cast(const) environment.toAA : 1216 env; 1217 1218 // fetching all the request headers 1219 string[string] requestHeadersHere; 1220 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1221 if(k.startsWith("HTTP_")) { 1222 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1223 } 1224 } 1225 1226 this.requestHeaders = assumeUnique(requestHeadersHere); 1227 1228 requestUri = getenv("REQUEST_URI"); 1229 1230 cookie = getenv("HTTP_COOKIE"); 1231 cookiesArray = getCookieArray(); 1232 cookies = keepLastOf(cookiesArray); 1233 1234 referrer = getenv("HTTP_REFERER"); 1235 userAgent = getenv("HTTP_USER_AGENT"); 1236 remoteAddress = getenv("REMOTE_ADDR"); 1237 host = getenv("HTTP_HOST"); 1238 pathInfo = getenv("PATH_INFO"); 1239 1240 queryString = getenv("QUERY_STRING"); 1241 scriptName = getenv("SCRIPT_NAME"); 1242 { 1243 import core.runtime; 1244 auto sfn = getenv("SCRIPT_FILENAME"); 1245 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1246 } 1247 1248 bool iis = false; 1249 1250 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1251 if(requestUri.length == 0) { 1252 // IIS sometimes includes the script name as part of the path info - we don't want that 1253 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1254 pathInfo = pathInfo[scriptName.length .. $]; 1255 1256 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1257 1258 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1259 1260 // FIXME: this works for apache and iis... but what about others? 1261 } 1262 1263 1264 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1265 getArray = assumeUnique(ugh); 1266 get = keepLastOf(getArray); 1267 1268 1269 // NOTE: on shitpache, you need to specifically forward this 1270 authorization = getenv("HTTP_AUTHORIZATION"); 1271 // this is a hack because Apache is a shitload of fuck and 1272 // refuses to send the real header to us. Compatible 1273 // programs should send both the standard and X- versions 1274 1275 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1276 // unnecessary with mod_rewrite, so it is commented 1277 1278 //if(authorization.length == 0) // if the std is there, use it 1279 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1280 1281 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1282 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1283 port = to!int(getenv("SERVER_PORT")); 1284 else 1285 port = 0; // this was probably called from the command line 1286 1287 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1288 if(ae.length && ae.indexOf("gzip") != -1) 1289 acceptsGzip = true; 1290 1291 accept = getenv("HTTP_ACCEPT"); 1292 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1293 1294 auto ka = getenv("HTTP_CONNECTION"); 1295 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1296 keepAliveRequested = true; 1297 1298 auto or = getenv("HTTP_ORIGIN"); 1299 origin = or; 1300 1301 auto rm = getenv("REQUEST_METHOD"); 1302 if(rm.length) 1303 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1304 else 1305 requestMethod = RequestMethod.CommandLine; 1306 1307 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1308 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1309 1310 // FIXME: DOCUMENT_ROOT? 1311 1312 // FIXME: what about PUT? 1313 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1314 version(preserveData) // a hack to make forwarding simpler 1315 immutable(ubyte)[] data; 1316 size_t amountReceived = 0; 1317 auto contentType = getenv("CONTENT_TYPE"); 1318 1319 // FIXME: is this ever not going to be set? I guess it depends 1320 // on if the server de-chunks and buffers... seems like it has potential 1321 // to be slow if they did that. The spec says it is always there though. 1322 // And it has worked reliably for me all year in the live environment, 1323 // but some servers might be different. 1324 auto cls = getenv("CONTENT_LENGTH"); 1325 auto contentLength = to!size_t(cls.length ? cls : "0"); 1326 1327 immutable originalContentLength = contentLength; 1328 if(contentLength) { 1329 if(maxContentLength > 0 && contentLength > maxContentLength) { 1330 setResponseStatus("413 Request entity too large"); 1331 write("You tried to upload a file that is too large."); 1332 close(); 1333 throw new Exception("POST too large"); 1334 } 1335 prepareForIncomingDataChunks(contentType, contentLength); 1336 1337 1338 int processChunk(in ubyte[] chunk) { 1339 if(chunk.length > contentLength) { 1340 handleIncomingDataChunk(chunk[0..contentLength]); 1341 amountReceived += contentLength; 1342 contentLength = 0; 1343 return 1; 1344 } else { 1345 handleIncomingDataChunk(chunk); 1346 contentLength -= chunk.length; 1347 amountReceived += chunk.length; 1348 } 1349 if(contentLength == 0) 1350 return 1; 1351 1352 onRequestBodyDataReceived(amountReceived, originalContentLength); 1353 return 0; 1354 } 1355 1356 1357 if(readdata is null) { 1358 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1359 if(processChunk(chunk)) 1360 break; 1361 } else { 1362 // we have a custom data source.. 1363 auto chunk = readdata(); 1364 while(chunk.length) { 1365 if(processChunk(chunk)) 1366 break; 1367 chunk = readdata(); 1368 } 1369 } 1370 1371 onRequestBodyDataReceived(amountReceived, originalContentLength); 1372 postArray = assumeUnique(pps._post); 1373 filesArray = assumeUnique(pps._files); 1374 files = keepLastOf(filesArray); 1375 post = keepLastOf(postArray); 1376 this.postBody = pps.postBody; 1377 cleanUpPostDataState(); 1378 } 1379 1380 version(preserveData) 1381 originalPostData = data; 1382 } 1383 // fixme: remote_user script name 1384 1385 1386 this.port = port; 1387 this.referrer = referrer; 1388 this.remoteAddress = remoteAddress; 1389 this.userAgent = userAgent; 1390 this.authorization = authorization; 1391 this.origin = origin; 1392 this.accept = accept; 1393 this.lastEventId = lastEventId; 1394 this.https = https; 1395 this.host = host; 1396 this.requestMethod = requestMethod; 1397 this.requestUri = requestUri; 1398 this.pathInfo = pathInfo; 1399 this.queryString = queryString; 1400 } 1401 1402 /// Cleans up any temporary files. Do not use the object 1403 /// after calling this. 1404 /// 1405 /// NOTE: it is called automatically by GenericMain 1406 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1407 void dispose() { 1408 foreach(file; files) { 1409 if(!file.contentInMemory) 1410 if(std.file.exists(file.contentFilename)) 1411 std.file.remove(file.contentFilename); 1412 } 1413 } 1414 1415 private { 1416 struct PostParserState { 1417 string contentType; 1418 string boundary; 1419 string localBoundary; // the ones used at the end or something lol 1420 bool isMultipart; 1421 bool needsSavedBody; 1422 1423 ulong expectedLength; 1424 ulong contentConsumed; 1425 immutable(ubyte)[] buffer; 1426 1427 // multipart parsing state 1428 int whatDoWeWant; 1429 bool weHaveAPart; 1430 string[] thisOnesHeaders; 1431 immutable(ubyte)[] thisOnesData; 1432 1433 string postBody; 1434 1435 UploadedFile piece; 1436 bool isFile = false; 1437 1438 size_t memoryCommitted; 1439 1440 // do NOT keep mutable references to these anywhere! 1441 // I assume they are unique in the constructor once we're all done getting data. 1442 string[][string] _post; 1443 UploadedFile[][string] _files; 1444 } 1445 1446 PostParserState pps; 1447 } 1448 1449 /// This represents a file the user uploaded via a POST request. 1450 static struct UploadedFile { 1451 /// If you want to create one of these structs for yourself from some data, 1452 /// use this function. 1453 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1454 Cgi.UploadedFile f; 1455 f.filename = name; 1456 f.content = cast(immutable(ubyte)[]) data; 1457 f.contentInMemory = true; 1458 return f; 1459 } 1460 1461 string name; /// The name of the form element. 1462 string filename; /// The filename the user set. 1463 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1464 1465 /** 1466 For small files, cgi.d will buffer the uploaded file in memory, and make it 1467 directly accessible to you through the content member. I find this very convenient 1468 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1469 often want to inspect and modify the file anyway!) 1470 1471 I find the file is very large, it is undesirable to eat that much memory just 1472 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1473 to the constructor so they are accepted, cgi.d will write the content to a temporary 1474 file that you can re-read later. 1475 1476 You can override this behavior by subclassing Cgi and overriding the protected 1477 handlePostChunk method. Note that the object is not initialized when you 1478 write that method - the http headers are available, but the cgi.post method 1479 is not. You may parse the file as it streams in using this method. 1480 1481 1482 Anyway, if the file is small enough to be in memory, contentInMemory will be 1483 set to true, and the content is available in the content member. 1484 1485 If not, contentInMemory will be set to false, and the content saved in a file, 1486 whose name will be available in the contentFilename member. 1487 1488 1489 Tip: if you know you are always dealing with small files, and want the convenience 1490 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1491 a large file comes in, it simply throws an exception (and HTTP error response) 1492 instead of trying to handle it. 1493 1494 The default value of maxContentLength in the constructor is for small files. 1495 */ 1496 bool contentInMemory = true; // the default ought to always be true 1497 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1498 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1499 1500 /// 1501 ulong fileSize() const { 1502 if(contentInMemory) 1503 return content.length; 1504 import std.file; 1505 return std.file.getSize(contentFilename); 1506 1507 } 1508 1509 /// 1510 void writeToFile(string filenameToSaveTo) const { 1511 import std.file; 1512 if(contentInMemory) 1513 std.file.write(filenameToSaveTo, content); 1514 else 1515 std.file.rename(contentFilename, filenameToSaveTo); 1516 } 1517 } 1518 1519 // given a content type and length, decide what we're going to do with the data.. 1520 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1521 pps.expectedLength = contentLength; 1522 1523 auto terminator = contentType.indexOf(";"); 1524 if(terminator == -1) 1525 terminator = contentType.length; 1526 1527 pps.contentType = contentType[0 .. terminator]; 1528 auto b = contentType[terminator .. $]; 1529 if(b.length) { 1530 auto idx = b.indexOf("boundary="); 1531 if(idx != -1) { 1532 pps.boundary = b[idx + "boundary=".length .. $]; 1533 pps.localBoundary = "\r\n--" ~ pps.boundary; 1534 } 1535 } 1536 1537 // while a content type SHOULD be sent according to the RFC, it is 1538 // not required. We're told we SHOULD guess by looking at the content 1539 // but it seems to me that this only happens when it is urlencoded. 1540 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1541 pps.isMultipart = false; 1542 pps.needsSavedBody = false; 1543 } else if(pps.contentType == "multipart/form-data") { 1544 pps.isMultipart = true; 1545 enforce(pps.boundary.length, "no boundary"); 1546 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1547 // save the body so the application can handle it 1548 pps.isMultipart = false; 1549 pps.needsSavedBody = true; 1550 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1551 // save the body so the application can handle it 1552 pps.needsSavedBody = true; 1553 pps.isMultipart = false; 1554 } else { 1555 // the rest is 100% handled by the application. just save the body and send it to them 1556 pps.needsSavedBody = true; 1557 pps.isMultipart = false; 1558 } 1559 } 1560 1561 // handles streaming POST data. If you handle some other content type, you should 1562 // override this. If the data isn't the content type you want, you ought to call 1563 // super.handleIncomingDataChunk so regular forms and files still work. 1564 1565 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1566 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1567 // input anyway, so I'm not going to get too worked up about it right now. 1568 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1569 if(chunk.length == 0) 1570 return; 1571 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1572 // if we're passed big chunks, it might throw unnecessarily. 1573 // just pass it smaller chunks at a time. 1574 if(pps.isMultipart) { 1575 // multipart/form-data 1576 1577 1578 // FIXME: this might want to be factored out and factorized 1579 // need to make sure the stream hooks actually work. 1580 void pieceHasNewContent() { 1581 // we just grew the piece's buffer. Do we have to switch to file backing? 1582 if(pps.piece.contentInMemory) { 1583 if(pps.piece.content.length <= 10 * 1024 * 1024) 1584 // meh, I'm ok with it. 1585 return; 1586 else { 1587 // this is too big. 1588 if(!pps.isFile) 1589 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1590 else { 1591 // a file this large is probably acceptable though... let's use a backing file. 1592 pps.piece.contentInMemory = false; 1593 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1594 1595 int count = 0; 1596 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1597 // odds are this loop will never be entered, but we want it just in case. 1598 while(std.file.exists(pps.piece.contentFilename)) { 1599 count++; 1600 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1601 } 1602 // I hope this creates the file pretty quickly, or the loop might be useless... 1603 // FIXME: maybe I should write some kind of custom transaction here. 1604 std.file.write(pps.piece.contentFilename, pps.piece.content); 1605 1606 pps.piece.content = null; 1607 } 1608 } 1609 } else { 1610 // it's already in a file, so just append it to what we have 1611 if(pps.piece.content.length) { 1612 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1613 std.file.append(pps.piece.contentFilename, pps.piece.content); 1614 pps.piece.content = null; 1615 } 1616 } 1617 } 1618 1619 1620 void commitPart() { 1621 if(!pps.weHaveAPart) 1622 return; 1623 1624 pieceHasNewContent(); // be sure the new content is handled every time 1625 1626 if(pps.isFile) { 1627 // I'm not sure if other environments put files in post or not... 1628 // I used to not do it, but I think I should, since it is there... 1629 pps._post[pps.piece.name] ~= pps.piece.filename; 1630 pps._files[pps.piece.name] ~= pps.piece; 1631 1632 allPostNamesInOrder ~= pps.piece.name; 1633 allPostValuesInOrder ~= pps.piece.filename; 1634 } else { 1635 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1636 1637 allPostNamesInOrder ~= pps.piece.name; 1638 allPostValuesInOrder ~= cast(string) pps.piece.content; 1639 } 1640 1641 /* 1642 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1643 pps.piece.content.length < 1000 1644 ? 1645 to!string(pps.piece.content) 1646 : 1647 "too long"); 1648 */ 1649 1650 // FIXME: the limit here 1651 pps.memoryCommitted += pps.piece.content.length; 1652 1653 pps.weHaveAPart = false; 1654 pps.whatDoWeWant = 1; 1655 pps.thisOnesHeaders = null; 1656 pps.thisOnesData = null; 1657 1658 pps.piece = UploadedFile.init; 1659 pps.isFile = false; 1660 } 1661 1662 void acceptChunk() { 1663 pps.buffer ~= chunk; 1664 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1665 } 1666 1667 immutable(ubyte)[] consume(size_t howMuch) { 1668 pps.contentConsumed += howMuch; 1669 auto ret = pps.buffer[0 .. howMuch]; 1670 pps.buffer = pps.buffer[howMuch .. $]; 1671 return ret; 1672 } 1673 1674 dataConsumptionLoop: do { 1675 switch(pps.whatDoWeWant) { 1676 default: assert(0); 1677 case 0: 1678 acceptChunk(); 1679 // the format begins with two extra leading dashes, then we should be at the boundary 1680 if(pps.buffer.length < 2) 1681 return; 1682 assert(pps.buffer[0] == '-', "no leading dash"); 1683 consume(1); 1684 assert(pps.buffer[0] == '-', "no second leading dash"); 1685 consume(1); 1686 1687 pps.whatDoWeWant = 1; 1688 goto case 1; 1689 /* fallthrough */ 1690 case 1: // looking for headers 1691 // here, we should be lined up right at the boundary, which is followed by a \r\n 1692 1693 // want to keep the buffer under control in case we're under attack 1694 //stderr.writeln("here once"); 1695 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1696 // throw new Exception("wtf is up with the huge mime part headers"); 1697 1698 acceptChunk(); 1699 1700 if(pps.buffer.length < pps.boundary.length) 1701 return; // not enough data, since there should always be a boundary here at least 1702 1703 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1704 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1705 // we *should* be at the end here! 1706 assert(pps.buffer[0] == '-'); 1707 consume(1); 1708 assert(pps.buffer[0] == '-'); 1709 consume(1); 1710 1711 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1712 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1713 "not lined up on boundary " ~ pps.boundary); 1714 consume(pps.boundary.length); 1715 1716 assert(pps.buffer[0] == '-'); 1717 consume(1); 1718 assert(pps.buffer[0] == '-'); 1719 consume(1); 1720 1721 assert(pps.buffer[0] == '\r'); 1722 consume(1); 1723 assert(pps.buffer[0] == '\n'); 1724 consume(1); 1725 1726 assert(pps.buffer.length == 0); 1727 assert(pps.contentConsumed == pps.expectedLength); 1728 break dataConsumptionLoop; // we're done! 1729 } else { 1730 // we're not done yet. We should be lined up on a boundary. 1731 1732 // But, we want to ensure the headers are here before we consume anything! 1733 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1734 if(headerEndLocation == -1) 1735 return; // they *should* all be here, so we can handle them all at once. 1736 1737 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1738 "not lined up on boundary " ~ pps.boundary); 1739 1740 consume(pps.boundary.length); 1741 // the boundary is always followed by a \r\n 1742 assert(pps.buffer[0] == '\r'); 1743 consume(1); 1744 assert(pps.buffer[0] == '\n'); 1745 consume(1); 1746 } 1747 1748 // re-running since by consuming the boundary, we invalidate the old index. 1749 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1750 assert(headerEndLocation >= 0, "no header"); 1751 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1752 1753 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1754 1755 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1756 1757 // now we'll parse the headers 1758 foreach(h; pps.thisOnesHeaders) { 1759 auto p = h.indexOf(":"); 1760 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1761 string hn = h[0..p]; 1762 string hv = h[p+2..$]; 1763 1764 switch(hn.toLower) { 1765 default: assert(0); 1766 case "content-disposition": 1767 auto info = hv.split("; "); 1768 foreach(i; info[1..$]) { // skipping the form-data 1769 auto o = i.split("="); // FIXME 1770 string pn = o[0]; 1771 string pv = o[1][1..$-1]; 1772 1773 if(pn == "name") { 1774 pps.piece.name = pv; 1775 } else if (pn == "filename") { 1776 pps.piece.filename = pv; 1777 pps.isFile = true; 1778 } 1779 } 1780 break; 1781 case "content-type": 1782 pps.piece.contentType = hv; 1783 break; 1784 } 1785 } 1786 1787 pps.whatDoWeWant++; // move to the next step - the data 1788 break; 1789 case 2: 1790 // when we get here, pps.buffer should contain our first chunk of data 1791 1792 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1793 throw new Exception("wtf is up with the huge mime part buffer"); 1794 1795 acceptChunk(); 1796 1797 // so the trick is, we want to process all the data up to the boundary, 1798 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1799 // want to wait for the next chunk. We start by looking for the whole boundary 1800 // in the buffer somewhere. 1801 1802 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1803 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1804 if(boundaryLocation != -1) { 1805 // this is easy - we can see it in it's entirety! 1806 1807 pps.piece.content ~= consume(boundaryLocation); 1808 1809 assert(pps.buffer[0] == '\r'); 1810 consume(1); 1811 assert(pps.buffer[0] == '\n'); 1812 consume(1); 1813 assert(pps.buffer[0] == '-'); 1814 consume(1); 1815 assert(pps.buffer[0] == '-'); 1816 consume(1); 1817 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1818 pps.weHaveAPart = true; 1819 pps.whatDoWeWant = 1; // back to getting headers for the next part 1820 1821 commitPart(); // we're done here 1822 } else { 1823 // we can't see the whole thing, but what if there's a partial boundary? 1824 1825 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1826 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1827 bool potentialBoundaryFound = false; 1828 1829 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1830 // we grow the boundary a bit each time. If we think it looks the 1831 // same, better pull another chunk to be sure it's not the end. 1832 // Starting small because exiting the loop early is desirable, since 1833 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1834 // the best we can do. 1835 if(a > pps.buffer.length) 1836 break; // FIXME: is this right? 1837 assert(a <= pps.buffer.length); 1838 assert(a > 0); 1839 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1840 // ok, there *might* be a boundary here, so let's 1841 // not treat the end as data yet. The rest is good to 1842 // use though, since if there was a boundary there, we'd 1843 // have handled it up above after locationOf. 1844 1845 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1846 consume(pps.buffer.length - a); 1847 pieceHasNewContent(); 1848 potentialBoundaryFound = true; 1849 break boundaryCheck; 1850 } 1851 } 1852 1853 if(!potentialBoundaryFound) { 1854 // we can consume the whole thing 1855 pps.piece.content ~= pps.buffer; 1856 pieceHasNewContent(); 1857 consume(pps.buffer.length); 1858 } else { 1859 // we found a possible boundary, but there was 1860 // insufficient data to be sure. 1861 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1862 1863 return; // wait for the next chunk. 1864 } 1865 } 1866 } 1867 } while(pps.buffer.length); 1868 1869 // btw all boundaries except the first should have a \r\n before them 1870 } else { 1871 // application/x-www-form-urlencoded and application/json 1872 1873 // not using maxContentLength because that might be cranked up to allow 1874 // large file uploads. We can handle them, but a huge post[] isn't any good. 1875 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1876 throw new Exception("wtf is up with such a gigantic form submission????"); 1877 1878 pps.buffer ~= chunk; 1879 1880 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1881 if(pps.buffer.length == pps.expectedLength) { 1882 if(pps.needsSavedBody) 1883 pps.postBody = cast(string) pps.buffer; 1884 else 1885 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1886 version(preserveData) 1887 originalPostData = pps.buffer; 1888 } else { 1889 // just for debugging 1890 } 1891 } 1892 } 1893 1894 protected void cleanUpPostDataState() { 1895 pps = PostParserState.init; 1896 } 1897 1898 /// you can override this function to somehow react 1899 /// to an upload in progress. 1900 /// 1901 /// Take note that parts of the CGI object is not yet 1902 /// initialized! Stuff from HTTP headers, including get[], is usable. 1903 /// But, none of post[] is usable, and you cannot write here. That's 1904 /// why this method is const - mutating the object won't do much anyway. 1905 /// 1906 /// My idea here was so you can output a progress bar or 1907 /// something to a cooperative client (see arsd.rtud for a potential helper) 1908 /// 1909 /// The default is to do nothing. Subclass cgi and use the 1910 /// CustomCgiMain mixin to do something here. 1911 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1912 // This space intentionally left blank. 1913 } 1914 1915 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1916 /// *closeConnection will be set to true if you should close the connection after handling this request 1917 this(BufferedInputRange ir, bool* closeConnection) { 1918 isCalledWithCommandLineArguments = false; 1919 import al = std.algorithm; 1920 1921 immutable(ubyte)[] data; 1922 1923 void rdo(const(ubyte)[] d) { 1924 //import std.stdio; writeln(d); 1925 sendAll(ir.source, d); 1926 } 1927 1928 auto ira = ir.source.remoteAddress(); 1929 auto irLocalAddress = ir.source.localAddress(); 1930 1931 ushort port = 80; 1932 if(auto ia = cast(InternetAddress) irLocalAddress) { 1933 port = ia.port; 1934 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1935 port = ia.port; 1936 } 1937 1938 // that check for UnixAddress is to work around a Phobos bug 1939 // see: https://github.com/dlang/phobos/pull/7383 1940 // but this might be more useful anyway tbh for this case 1941 version(Posix) 1942 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1943 else 1944 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1945 } 1946 1947 /** 1948 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1949 1950 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1951 1952 Params: 1953 inputData = the incoming data, including headers and other raw http data. 1954 When the constructor exits, it will leave this range exactly at the start of 1955 the next request on the connection (if there is one). 1956 1957 address = the IP address of the remote user 1958 _port = the port number of the connection 1959 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1960 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1961 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1962 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1963 closeConnection = if the request asks to close the connection, *closeConnection == true. 1964 */ 1965 this( 1966 BufferedInputRange inputData, 1967 // string[] headers, immutable(ubyte)[] data, 1968 string address, ushort _port, 1969 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1970 bool _https = false, 1971 void delegate(const(ubyte)[]) _rawDataOutput = null, 1972 void delegate() _flush = null, 1973 // this pointer tells if the connection is supposed to be closed after we handle this 1974 bool* closeConnection = null) 1975 { 1976 // these are all set locally so the loop works 1977 // without triggering errors in dmd 2.064 1978 // we go ahead and set them at the end of it to the this version 1979 int port; 1980 string referrer; 1981 string remoteAddress; 1982 string userAgent; 1983 string authorization; 1984 string origin; 1985 string accept; 1986 string lastEventId; 1987 bool https; 1988 string host; 1989 RequestMethod requestMethod; 1990 string requestUri; 1991 string pathInfo; 1992 string queryString; 1993 string scriptName; 1994 string[string] get; 1995 string[][string] getArray; 1996 bool keepAliveRequested; 1997 bool acceptsGzip; 1998 string cookie; 1999 2000 2001 2002 environmentVariables = cast(const) environment.toAA; 2003 2004 idlol = inputData; 2005 2006 isCalledWithCommandLineArguments = false; 2007 2008 https = _https; 2009 port = _port; 2010 2011 rawDataOutput = _rawDataOutput; 2012 flushDelegate = _flush; 2013 nph = true; 2014 2015 remoteAddress = address; 2016 2017 // streaming parser 2018 import al = std.algorithm; 2019 2020 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 2021 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 2022 while(idx == -1) { 2023 inputData.popFront(0); 2024 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 2025 } 2026 2027 assert(idx != -1); 2028 2029 2030 string contentType = ""; 2031 string[string] requestHeadersHere; 2032 2033 size_t contentLength; 2034 2035 bool isChunked; 2036 2037 { 2038 import core.runtime; 2039 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 2040 } 2041 2042 2043 int headerNumber = 0; 2044 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 2045 if(line.length) { 2046 headerNumber++; 2047 auto header = cast(string) line.idup; 2048 if(headerNumber == 1) { 2049 // request line 2050 auto parts = al.splitter(header, " "); 2051 if(parts.front == "PRI") { 2052 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 2053 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 2054 // to bring me benefit) 2055 throw new HttpVersionNotSupportedException(); 2056 } 2057 requestMethod = to!RequestMethod(parts.front); 2058 parts.popFront(); 2059 requestUri = parts.front; 2060 2061 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 2062 scriptName = requestUri[0 .. pathInfoStarts]; 2063 2064 auto question = requestUri.indexOf("?"); 2065 if(question == -1) { 2066 queryString = ""; 2067 // FIXME: double check, this might be wrong since it could be url encoded 2068 pathInfo = requestUri[pathInfoStarts..$]; 2069 } else { 2070 queryString = requestUri[question+1..$]; 2071 pathInfo = requestUri[pathInfoStarts..question]; 2072 } 2073 2074 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2075 getArray = cast(string[][string]) assumeUnique(ugh); 2076 2077 if(header.indexOf("HTTP/1.0") != -1) { 2078 http10 = true; 2079 autoBuffer = true; 2080 if(closeConnection) { 2081 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2082 *closeConnection = true; 2083 } 2084 } 2085 } else { 2086 // other header 2087 auto colon = header.indexOf(":"); 2088 if(colon == -1) 2089 throw new Exception("HTTP headers should have a colon!"); 2090 string name = header[0..colon].toLower; 2091 string value = header[colon+2..$]; // skip the colon and the space 2092 2093 requestHeadersHere[name] = value; 2094 2095 if (name == "accept") { 2096 accept = value; 2097 } 2098 else if (name == "origin") { 2099 origin = value; 2100 } 2101 else if (name == "connection") { 2102 if(value == "close" && closeConnection) 2103 *closeConnection = true; 2104 if(value.asLowerCase().canFind("keep-alive")) { 2105 keepAliveRequested = true; 2106 2107 // on http 1.0, the connection is closed by default, 2108 // but not if they request keep-alive. then we don't close 2109 // anymore - undoing the set above 2110 if(http10 && closeConnection) { 2111 *closeConnection = false; 2112 } 2113 } 2114 } 2115 else if (name == "transfer-encoding") { 2116 if(value == "chunked") 2117 isChunked = true; 2118 } 2119 else if (name == "last-event-id") { 2120 lastEventId = value; 2121 } 2122 else if (name == "authorization") { 2123 authorization = value; 2124 } 2125 else if (name == "content-type") { 2126 contentType = value; 2127 } 2128 else if (name == "content-length") { 2129 contentLength = to!size_t(value); 2130 } 2131 else if (name == "x-forwarded-for") { 2132 remoteAddress = value; 2133 } 2134 else if (name == "x-forwarded-host" || name == "host") { 2135 if(name != "host" || host is null) 2136 host = value; 2137 } 2138 // FIXME: https://tools.ietf.org/html/rfc7239 2139 else if (name == "accept-encoding") { 2140 if(value.indexOf("gzip") != -1) 2141 acceptsGzip = true; 2142 } 2143 else if (name == "user-agent") { 2144 userAgent = value; 2145 } 2146 else if (name == "referer") { 2147 referrer = value; 2148 } 2149 else if (name == "cookie") { 2150 cookie ~= value; 2151 } else if(name == "expect") { 2152 if(value == "100-continue") { 2153 // FIXME we should probably give user code a chance 2154 // to process and reject but that needs to be virtual, 2155 // perhaps part of the CGI redesign. 2156 2157 // FIXME: if size is > max content length it should 2158 // also fail at this point. 2159 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2160 2161 // FIXME: let the user write out 103 early hints too 2162 } 2163 } 2164 // else 2165 // ignore it 2166 2167 } 2168 } 2169 2170 inputData.consume(idx + 4); 2171 // done 2172 2173 requestHeaders = assumeUnique(requestHeadersHere); 2174 2175 ByChunkRange dataByChunk; 2176 2177 // reading Content-Length type data 2178 // We need to read up the data we have, and write it out as a chunk. 2179 if(!isChunked) { 2180 dataByChunk = byChunk(inputData, contentLength); 2181 } else { 2182 // chunked requests happen, but not every day. Since we need to know 2183 // the content length (for now, maybe that should change), we'll buffer 2184 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2185 auto data = dechunk(inputData); 2186 2187 // set the range here 2188 dataByChunk = byChunk(data); 2189 contentLength = data.length; 2190 } 2191 2192 assert(dataByChunk !is null); 2193 2194 if(contentLength) { 2195 prepareForIncomingDataChunks(contentType, contentLength); 2196 foreach(dataChunk; dataByChunk) { 2197 handleIncomingDataChunk(dataChunk); 2198 } 2199 postArray = assumeUnique(pps._post); 2200 filesArray = assumeUnique(pps._files); 2201 files = keepLastOf(filesArray); 2202 post = keepLastOf(postArray); 2203 postBody = pps.postBody; 2204 cleanUpPostDataState(); 2205 } 2206 2207 this.port = port; 2208 this.referrer = referrer; 2209 this.remoteAddress = remoteAddress; 2210 this.userAgent = userAgent; 2211 this.authorization = authorization; 2212 this.origin = origin; 2213 this.accept = accept; 2214 this.lastEventId = lastEventId; 2215 this.https = https; 2216 this.host = host; 2217 this.requestMethod = requestMethod; 2218 this.requestUri = requestUri; 2219 this.pathInfo = pathInfo; 2220 this.queryString = queryString; 2221 2222 this.scriptName = scriptName; 2223 this.get = keepLastOf(getArray); 2224 this.getArray = cast(immutable) getArray; 2225 this.keepAliveRequested = keepAliveRequested; 2226 this.acceptsGzip = acceptsGzip; 2227 this.cookie = cookie; 2228 2229 cookiesArray = getCookieArray(); 2230 cookies = keepLastOf(cookiesArray); 2231 2232 } 2233 BufferedInputRange idlol; 2234 2235 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2236 string[string] ca; 2237 foreach(k, v; arr) 2238 ca[k] = v[$-1]; 2239 2240 return assumeUnique(ca); 2241 } 2242 2243 // FIXME duplication 2244 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2245 UploadedFile[string] ca; 2246 foreach(k, v; arr) 2247 ca[k] = v[$-1]; 2248 2249 return assumeUnique(ca); 2250 } 2251 2252 2253 private immutable(string[][string]) getCookieArray() { 2254 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2255 return assumeUnique(forTheLoveOfGod); 2256 } 2257 2258 /++ 2259 Very simple method to require a basic auth username and password. 2260 If the http request doesn't include the required credentials, it throws a 2261 HTTP 401 error, and an exception to cancel your handler. Do NOT catch the 2262 `AuthorizationRequiredException` exception thrown by this if you want the 2263 http basic auth prompt to work for the user! 2264 2265 Note: basic auth does not provide great security, especially over unencrypted HTTP; 2266 the user's credentials are sent in plain text on every request. 2267 2268 If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2269 application. Either use Apache's built in methods for basic authentication, or add 2270 something along these lines to your server configuration: 2271 2272 ``` 2273 RewriteEngine On 2274 RewriteCond %{HTTP:Authorization} ^(.*) 2275 RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2276 ``` 2277 2278 To ensure the necessary data is available to cgi.d. 2279 +/ 2280 void requireBasicAuth(string user, string pass, string message = null, string file = __FILE__, size_t line = __LINE__) { 2281 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2282 throw new AuthorizationRequiredException("Basic", message, file, line); 2283 } 2284 } 2285 2286 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2287 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2288 /// Use setResponseExpires and updateResponseExpires for more control 2289 void setCache(bool allowCaching) { 2290 noCache = !allowCaching; 2291 } 2292 2293 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2294 /// who can accept it 2295 bool gzipResponse; 2296 2297 immutable bool acceptsGzip; 2298 immutable bool keepAliveRequested; 2299 2300 /// Set to true if and only if this was initialized with command line arguments 2301 immutable bool isCalledWithCommandLineArguments; 2302 2303 /// This gets a full url for the current request, including port, protocol, host, path, and query 2304 string getCurrentCompleteUri() const { 2305 ushort defaultPort = https ? 443 : 80; 2306 2307 string uri = "http"; 2308 if(https) 2309 uri ~= "s"; 2310 uri ~= "://"; 2311 uri ~= host; 2312 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2313 version(none) 2314 if(!(!port || port == defaultPort)) { 2315 uri ~= ":"; 2316 uri ~= to!string(port); 2317 } 2318 +/ 2319 uri ~= requestUri; 2320 return uri; 2321 } 2322 2323 /// You can override this if your site base url isn't the same as the script name 2324 string logicalScriptName() const { 2325 return scriptName; 2326 } 2327 2328 /++ 2329 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2330 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2331 Note setResponseStatus() must be called *before* you write() any data to the output. 2332 2333 History: 2334 The `int` overload was added on January 11, 2021. 2335 +/ 2336 void setResponseStatus(string status) { 2337 assert(!outputtedResponseData); 2338 responseStatus = status; 2339 } 2340 /// ditto 2341 void setResponseStatus(int statusCode) { 2342 setResponseStatus(getHttpCodeText(statusCode)); 2343 } 2344 private string responseStatus = null; 2345 2346 /// Returns true if it is still possible to output headers 2347 bool canOutputHeaders() { 2348 return !isClosed && !outputtedResponseData; 2349 } 2350 2351 /// Sets the location header, which the browser will redirect the user to automatically. 2352 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2353 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2354 void setResponseLocation(string uri, bool important = true, string status = null) { 2355 if(!important && isCurrentResponseLocationImportant) 2356 return; // important redirects always override unimportant ones 2357 2358 if(uri is null) { 2359 responseStatus = "200 OK"; 2360 responseLocation = null; 2361 isCurrentResponseLocationImportant = important; 2362 return; // this just cancels the redirect 2363 } 2364 2365 assert(!outputtedResponseData); 2366 if(status is null) 2367 responseStatus = "302 Found"; 2368 else 2369 responseStatus = status; 2370 2371 responseLocation = uri.strip; 2372 isCurrentResponseLocationImportant = important; 2373 } 2374 protected string responseLocation = null; 2375 private bool isCurrentResponseLocationImportant = false; 2376 2377 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2378 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2379 /// Note: the when parameter is different than setCookie's expire parameter. 2380 void setResponseExpires(long when, bool isPublic = false) { 2381 responseExpires = when; 2382 setCache(true); // need to enable caching so the date has meaning 2383 2384 responseIsPublic = isPublic; 2385 responseExpiresRelative = false; 2386 } 2387 2388 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2389 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2390 responseExpires = whenFromNow; 2391 setCache(true); // need to enable caching so the date has meaning 2392 2393 responseIsPublic = isPublic; 2394 responseExpiresRelative = true; 2395 } 2396 private long responseExpires = long.min; 2397 private bool responseIsPublic = false; 2398 private bool responseExpiresRelative = false; 2399 2400 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2401 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2402 /// output as a whole is as cacheable as the least cachable part in the chain. 2403 2404 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2405 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2406 void updateResponseExpires(long when, bool isPublic) { 2407 if(responseExpires == long.min) 2408 setResponseExpires(when, isPublic); 2409 else if(when < responseExpires) 2410 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2411 } 2412 2413 /* 2414 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2415 /// Should generally be false if the user is logged in. It assumes private cache only. 2416 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2417 void setPublicCaching(bool allowPublicCaches) { 2418 publicCaching = allowPublicCaches; 2419 } 2420 private bool publicCaching = false; 2421 */ 2422 2423 /++ 2424 History: 2425 Added January 11, 2021 2426 +/ 2427 enum SameSitePolicy { 2428 Lax, 2429 Strict, 2430 None 2431 } 2432 2433 /++ 2434 Sets an HTTP cookie, automatically encoding the data to the correct string. 2435 expiresIn is how many milliseconds in the future the cookie will expire. 2436 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2437 Note setCookie() must be called *before* you write() any data to the output. 2438 2439 History: 2440 Parameter `sameSitePolicy` was added on January 11, 2021. 2441 +/ 2442 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2443 assert(!outputtedResponseData); 2444 string cookie = std.uri.encodeComponent(name) ~ "="; 2445 cookie ~= std.uri.encodeComponent(data); 2446 if(path !is null) 2447 cookie ~= "; path=" ~ path; 2448 // FIXME: should I just be using max-age here? (also in cache below) 2449 if(expiresIn != 0) 2450 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2451 if(domain !is null) 2452 cookie ~= "; domain=" ~ domain; 2453 if(secure == true) 2454 cookie ~= "; Secure"; 2455 if(httpOnly == true ) 2456 cookie ~= "; HttpOnly"; 2457 final switch(sameSitePolicy) { 2458 case SameSitePolicy.Lax: 2459 cookie ~= "; SameSite=Lax"; 2460 break; 2461 case SameSitePolicy.Strict: 2462 cookie ~= "; SameSite=Strict"; 2463 break; 2464 case SameSitePolicy.None: 2465 cookie ~= "; SameSite=None"; 2466 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2467 break; 2468 } 2469 2470 if(auto idx = name in cookieIndexes) { 2471 responseCookies[*idx] = cookie; 2472 } else { 2473 cookieIndexes[name] = responseCookies.length; 2474 responseCookies ~= cookie; 2475 } 2476 } 2477 private string[] responseCookies; 2478 private size_t[string] cookieIndexes; 2479 2480 /// Clears a previously set cookie with the given name, path, and domain. 2481 void clearCookie(string name, string path = null, string domain = null) { 2482 assert(!outputtedResponseData); 2483 setCookie(name, "", 1, path, domain); 2484 } 2485 2486 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2487 void setResponseContentType(string ct) { 2488 assert(!outputtedResponseData); 2489 responseContentType = ct; 2490 } 2491 private string responseContentType = null; 2492 2493 /// Adds a custom header. It should be the name: value, but without any line terminator. 2494 /// For example: header("X-My-Header: Some value"); 2495 /// Note you should use the specialized functions in this object if possible to avoid 2496 /// duplicates in the output. 2497 void header(string h) { 2498 customHeaders ~= h; 2499 } 2500 2501 /++ 2502 I named the original function `header` after PHP, but this pattern more fits 2503 the rest of the Cgi object. 2504 2505 Either name are allowed. 2506 2507 History: 2508 Alias added June 17, 2022. 2509 +/ 2510 alias setResponseHeader = header; 2511 2512 private string[] customHeaders; 2513 private bool websocketMode; 2514 2515 void flushHeaders(const(void)[] t, bool isAll = false) { 2516 StackBuffer buffer = StackBuffer(0); 2517 2518 prepHeaders(t, isAll, &buffer); 2519 2520 if(rawDataOutput !is null) 2521 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2522 else { 2523 stdout.rawWrite(buffer.get()); 2524 } 2525 } 2526 2527 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2528 string terminator = "\n"; 2529 if(rawDataOutput !is null) 2530 terminator = "\r\n"; 2531 2532 if(responseStatus !is null) { 2533 if(nph) { 2534 if(http10) 2535 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2536 else 2537 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2538 } else 2539 buffer.add("Status: ", responseStatus, terminator); 2540 } else if (nph) { 2541 if(http10) 2542 buffer.add("HTTP/1.0 200 OK", terminator); 2543 else 2544 buffer.add("HTTP/1.1 200 OK", terminator); 2545 } 2546 2547 if(websocketMode) 2548 goto websocket; 2549 2550 if(nph) { // we're responsible for setting the date too according to http 1.1 2551 char[29] db = void; 2552 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2553 buffer.add("Date: ", db[], terminator); 2554 } 2555 2556 // FIXME: what if the user wants to set his own content-length? 2557 // The custom header function can do it, so maybe that's best. 2558 // Or we could reuse the isAll param. 2559 if(responseLocation !is null) { 2560 buffer.add("Location: ", responseLocation, terminator); 2561 } 2562 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2563 if(responseExpiresRelative) { 2564 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2565 buffer.add(responseExpires); 2566 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2567 } else { 2568 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2569 char[29] db = void; 2570 printDateToBuffer(cast(DateTime) expires, db[]); 2571 buffer.add("Expires: ", db[], terminator); 2572 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2573 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2574 buffer.add(terminator); 2575 } 2576 } 2577 if(responseCookies !is null && responseCookies.length > 0) { 2578 foreach(c; responseCookies) 2579 buffer.add("Set-Cookie: ", c, terminator); 2580 } 2581 if(noCache) { // we specifically do not want caching (this is actually the default) 2582 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2583 buffer.add("Expires: 0", terminator); 2584 buffer.add("Pragma: no-cache", terminator); 2585 } else { 2586 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2587 buffer.add("Cache-Control: public", terminator); 2588 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2589 } 2590 } 2591 if(responseContentType !is null) { 2592 buffer.add("Content-Type: ", responseContentType, terminator); 2593 } else 2594 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2595 2596 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2597 buffer.add("Content-Encoding: gzip", terminator); 2598 } 2599 2600 2601 if(!isAll) { 2602 if(nph && !http10) { 2603 buffer.add("Transfer-Encoding: chunked", terminator); 2604 responseChunked = true; 2605 } 2606 } else { 2607 buffer.add("Content-Length: "); 2608 buffer.add(t.length); 2609 buffer.add(terminator); 2610 if(nph && keepAliveRequested) { 2611 buffer.add("Connection: Keep-Alive", terminator); 2612 } 2613 } 2614 2615 websocket: 2616 2617 foreach(hd; customHeaders) 2618 buffer.add(hd, terminator); 2619 2620 // FIXME: what about duplicated headers? 2621 2622 // end of header indicator 2623 buffer.add(terminator); 2624 2625 outputtedResponseData = true; 2626 } 2627 2628 /// Writes the data to the output, flushing headers if they have not yet been sent. 2629 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2630 assert(!closed, "Output has already been closed"); 2631 2632 StackBuffer buffer = StackBuffer(0); 2633 2634 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2635 // actually gzip the data here 2636 2637 auto c = new Compress(HeaderFormat.gzip); // want gzip 2638 2639 auto data = c.compress(t); 2640 data ~= c.flush(); 2641 2642 // std.file.write("/tmp/last-item", data); 2643 2644 t = data; 2645 } 2646 2647 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2648 prepHeaders(t, isAll, &buffer); 2649 } 2650 2651 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2652 if (autoBuffer && !isAll) { 2653 outputBuffer ~= cast(ubyte[]) t; 2654 } 2655 if(!autoBuffer || isAll) { 2656 if(rawDataOutput !is null) 2657 if(nph && responseChunked) { 2658 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2659 // we're making the chunk here instead of in a function 2660 // to avoid unneeded gc pressure 2661 buffer.add(toHex(t.length)); 2662 buffer.add("\r\n"); 2663 buffer.add(cast(char[]) t, "\r\n"); 2664 } else { 2665 buffer.add(cast(char[]) t); 2666 } 2667 else 2668 buffer.add(cast(char[]) t); 2669 } 2670 } 2671 2672 if(rawDataOutput !is null) 2673 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2674 else 2675 stdout.rawWrite(buffer.get()); 2676 2677 if(maybeAutoClose && isAll) 2678 close(); // if you say it is all, that means we're definitely done 2679 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2680 } 2681 2682 /++ 2683 Convenience method to set content type to json and write the string as the complete response. 2684 2685 History: 2686 Added January 16, 2020 2687 +/ 2688 void writeJson(string json) { 2689 this.setResponseContentType("application/json"); 2690 this.write(json, true); 2691 } 2692 2693 /// Flushes the pending buffer, leaving the connection open so you can send more. 2694 void flush() { 2695 if(rawDataOutput is null) 2696 stdout.flush(); 2697 else if(flushDelegate !is null) 2698 flushDelegate(); 2699 } 2700 2701 version(autoBuffer) 2702 bool autoBuffer = true; 2703 else 2704 bool autoBuffer = false; 2705 ubyte[] outputBuffer; 2706 2707 /// Flushes the buffers to the network, signifying that you are done. 2708 /// You should always call this explicitly when you are done outputting data. 2709 void close() { 2710 if(closed) 2711 return; // don't double close 2712 2713 if(!outputtedResponseData) 2714 write("", true, false); 2715 2716 // writing auto buffered data 2717 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2718 if(!nph) 2719 stdout.rawWrite(outputBuffer); 2720 else 2721 write(outputBuffer, true, false); // tell it this is everything 2722 } 2723 2724 // closing the last chunk... 2725 if(nph && rawDataOutput !is null && responseChunked) 2726 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2727 2728 if(flushDelegate) 2729 flushDelegate(); 2730 2731 closed = true; 2732 } 2733 2734 // Closes without doing anything, shouldn't be used often 2735 void rawClose() { 2736 closed = true; 2737 } 2738 2739 /++ 2740 Gets a request variable as a specific type, or the default value of it isn't there 2741 or isn't convertible to the request type. 2742 2743 Checks both GET and POST variables, preferring the POST variable, if available. 2744 2745 A nice trick is using the default value to choose the type: 2746 2747 --- 2748 /* 2749 The return value will match the type of the default. 2750 Here, I gave 10 as a default, so the return value will 2751 be an int. 2752 2753 If the user-supplied value cannot be converted to the 2754 requested type, you will get the default value back. 2755 */ 2756 int a = cgi.request("number", 10); 2757 2758 if(cgi.get["number"] == "11") 2759 assert(a == 11); // conversion succeeds 2760 2761 if("number" !in cgi.get) 2762 assert(a == 10); // no value means you can't convert - give the default 2763 2764 if(cgi.get["number"] == "twelve") 2765 assert(a == 10); // conversion from string to int would fail, so we get the default 2766 --- 2767 2768 You can use an enum as an easy whitelist, too: 2769 2770 --- 2771 enum Operations { 2772 add, remove, query 2773 } 2774 2775 auto op = cgi.request("op", Operations.query); 2776 2777 if(cgi.get["op"] == "add") 2778 assert(op == Operations.add); 2779 if(cgi.get["op"] == "remove") 2780 assert(op == Operations.remove); 2781 if(cgi.get["op"] == "query") 2782 assert(op == Operations.query); 2783 2784 if(cgi.get["op"] == "random string") 2785 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2786 --- 2787 +/ 2788 T request(T = string)(in string name, in T def = T.init) const nothrow { 2789 try { 2790 return 2791 (name in post) ? to!T(post[name]) : 2792 (name in get) ? to!T(get[name]) : 2793 def; 2794 } catch(Exception e) { return def; } 2795 } 2796 2797 /// Is the output already closed? 2798 bool isClosed() const { 2799 return closed; 2800 } 2801 2802 private SessionObject commandLineSessionObject; 2803 2804 /++ 2805 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2806 +/ 2807 Session!Data getSessionObject(Data)() { 2808 if(testInProcess !is null) { 2809 // test mode 2810 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2811 if(obj !is null) 2812 return cast(typeof(return)) obj; 2813 else { 2814 auto o = new MockSession!Data(); 2815 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2816 return o; 2817 } 2818 } else { 2819 // FIXME: the changes are not printed out at the end! 2820 if(_commandLineSession !is null) { 2821 if(commandLineSessionObject is null) { 2822 auto clso = new MockSession!Data(); 2823 commandLineSessionObject = clso; 2824 2825 2826 foreach(memberName; __traits(allMembers, Data)) { 2827 if(auto str = memberName in _commandLineSession) 2828 __traits(getMember, clso.store_, memberName) = to!(typeof(__traits(getMember, Data, memberName)))(*str); 2829 } 2830 } 2831 2832 return cast(typeof(return)) commandLineSessionObject; 2833 } 2834 2835 // normal operation 2836 return new BasicDataServerSession!Data(this); 2837 } 2838 } 2839 2840 // if it is in test mode; triggers mock sessions. Used by CgiTester 2841 version(with_breaking_cgi_features) 2842 private CgiTester testInProcess; 2843 2844 /* Hooks for redirecting input and output */ 2845 private void delegate(const(ubyte)[]) rawDataOutput = null; 2846 private void delegate() flushDelegate = null; 2847 2848 /* This info is used when handling a more raw HTTP protocol */ 2849 private bool nph; 2850 private bool http10; 2851 private bool closed; 2852 private bool responseChunked = false; 2853 2854 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2855 immutable(ubyte)[] originalPostData; 2856 2857 /++ 2858 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2859 2860 It is intended to be used for JSON and XML request content types, but also may be used 2861 for other content types your application can handle. But it will NOT be populated 2862 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2863 parsed into the post and postArray members. 2864 2865 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2866 will be discarded to the client with an error. This helps keep this array from being exploded in size 2867 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2868 client in certain build modes.) 2869 2870 History: 2871 Added January 5, 2021 2872 Documented February 21, 2023 (dub v11.0) 2873 +/ 2874 public immutable string postBody; 2875 alias postJson = postBody; // old name 2876 2877 /* Internal state flags */ 2878 private bool outputtedResponseData; 2879 private bool noCache = true; 2880 2881 const(string[string]) environmentVariables; 2882 2883 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2884 partially because it logically is (your code doesn't change what the user requested...) 2885 and partially because I hate how bad programs in PHP change those superglobals to do 2886 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2887 2888 For some of these, you'll want to refer to the http or cgi specs for more details. 2889 */ 2890 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2891 2892 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2893 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2894 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2895 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2896 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2897 immutable(char[]) scriptFileName; /// The physical filename of your script 2898 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2899 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2900 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2901 2902 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2903 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2904 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2905 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2906 2907 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2908 */ 2909 immutable(char[]) referrer; 2910 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2911 2912 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2913 2914 immutable bool https; /// Was the request encrypted via https? 2915 immutable int port; /// On what TCP port number did the server receive the request? 2916 2917 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2918 2919 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2920 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2921 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2922 2923 /// added later 2924 alias query = get; 2925 2926 /** 2927 Represents user uploaded files. 2928 2929 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2930 */ 2931 immutable(UploadedFile[][string]) filesArray; 2932 immutable(UploadedFile[string]) files; 2933 2934 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2935 /// the order of the arrays is the order the data arrives 2936 immutable(string[][string]) getArray; /// like get, but an array of values per name 2937 immutable(string[][string]) postArray; /// ditto for post 2938 immutable(string[][string]) cookiesArray; /// ditto for cookies 2939 2940 private string[string] _commandLineSession; 2941 2942 // convenience function for appending to a uri without extra ? 2943 // matches the name and effect of javascript's location.search property 2944 string search() const { 2945 if(queryString.length) 2946 return "?" ~ queryString; 2947 return ""; 2948 } 2949 2950 // FIXME: what about multiple files with the same name? 2951 private: 2952 //RequestMethod _requestMethod; 2953 } 2954 2955 /// use this for testing or other isolated things when you want it to be no-ops 2956 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2957 // we want to ignore, not use stdout 2958 if(outputSink is null) 2959 outputSink = delegate void(const(ubyte)[]) { }; 2960 2961 string[string] env; 2962 env["REQUEST_METHOD"] = to!string(method); 2963 env["CONTENT_LENGTH"] = to!string(data.length); 2964 2965 auto cgi = new Cgi( 2966 0, 2967 env, 2968 { return data; }, 2969 outputSink, 2970 null); 2971 2972 return cgi; 2973 } 2974 2975 /++ 2976 A helper test class for request handler unittests. 2977 +/ 2978 version(with_breaking_cgi_features) 2979 class CgiTester { 2980 private { 2981 SessionObject[TypeInfo] mockSessions; 2982 SessionObject getSessionOverride(TypeInfo ti) { 2983 if(auto o = ti in mockSessions) 2984 return *o; 2985 else 2986 return null; 2987 } 2988 void setSessionOverride(TypeInfo ti, SessionObject so) { 2989 mockSessions[ti] = so; 2990 } 2991 } 2992 2993 /++ 2994 Gets (and creates if necessary) a mock session object for this test. Note 2995 it will be the same one used for any test operations through this CgiTester instance. 2996 +/ 2997 Session!Data getSessionObject(Data)() { 2998 auto obj = getSessionOverride(typeid(typeof(return))); 2999 if(obj !is null) 3000 return cast(typeof(return)) obj; 3001 else { 3002 auto o = new MockSession!Data(); 3003 setSessionOverride(typeid(typeof(return)), o); 3004 return o; 3005 } 3006 } 3007 3008 /++ 3009 Pass a reference to your request handler when creating the tester. 3010 +/ 3011 this(void function(Cgi) requestHandler) { 3012 this.requestHandler = requestHandler; 3013 } 3014 3015 /++ 3016 You can check response information with these methods after you call the request handler. 3017 +/ 3018 struct Response { 3019 int code; 3020 string[string] headers; 3021 string responseText; 3022 ubyte[] responseBody; 3023 } 3024 3025 /++ 3026 Executes a test request on your request handler, and returns the response. 3027 3028 Params: 3029 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 3030 args = additional arguments. Same format as cgi's command line handler. 3031 +/ 3032 Response GET(string url, string[] args = null) { 3033 return executeTest("GET", url, args); 3034 } 3035 /// ditto 3036 Response POST(string url, string[] args = null) { 3037 return executeTest("POST", url, args); 3038 } 3039 3040 /// ditto 3041 Response executeTest(string method, string url, string[] args) { 3042 ubyte[] outputtedRawData; 3043 void outputSink(const(ubyte)[] data) { 3044 outputtedRawData ~= data; 3045 } 3046 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 3047 cgi.testInProcess = this; 3048 scope(exit) cgi.dispose(); 3049 3050 requestHandler(cgi); 3051 3052 cgi.close(); 3053 3054 Response response; 3055 3056 if(outputtedRawData.length) { 3057 enum LINE = "\r\n"; 3058 3059 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 3060 assert(idx != -1, to!string(outputtedRawData)); 3061 auto headers = cast(string) outputtedRawData[0 .. idx]; 3062 response.code = 200; 3063 while(headers.length) { 3064 auto i = headers.locationOf(LINE); 3065 if(i == -1) i = cast(int) headers.length; 3066 3067 auto header = headers[0 .. i]; 3068 3069 auto c = header.locationOf(":"); 3070 if(c != -1) { 3071 auto name = header[0 .. c]; 3072 auto value = header[c + 2 ..$]; 3073 3074 if(name == "Status") 3075 response.code = value[0 .. value.locationOf(" ")].to!int; 3076 3077 response.headers[name] = value; 3078 } else { 3079 assert(0); 3080 } 3081 3082 if(i != headers.length) 3083 i += 2; 3084 headers = headers[i .. $]; 3085 } 3086 response.responseBody = outputtedRawData[idx + 4 .. $]; 3087 response.responseText = cast(string) response.responseBody; 3088 } 3089 3090 return response; 3091 } 3092 3093 private void function(Cgi) requestHandler; 3094 } 3095 3096 3097 // should this be a separate module? Probably, but that's a hassle. 3098 3099 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3100 string makeDataUrl(string mimeType, in void[] data) { 3101 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3102 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3103 } 3104 3105 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3106 /// Represents a url that can be broken down or built up through properties 3107 struct Uri { 3108 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3109 3110 // scheme//userinfo@host:port/path?query#fragment 3111 3112 string scheme; /// e.g. "http" in "http://example.com/" 3113 string userinfo; /// the username (and possibly a password) in the uri 3114 string host; /// the domain name 3115 int port; /// port number, if given. Will be zero if a port was not explicitly given 3116 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3117 string query; /// the stuff after the ? in a uri 3118 string fragment; /// the stuff after the # in a uri. 3119 3120 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3121 // the decode ones need to keep different names anyway because we can't overload on return values... 3122 static string encode(string s) { return std.uri.encodeComponent(s); } 3123 static string encode(string[string] s) { return encodeVariables(s); } 3124 static string encode(string[][string] s) { return encodeVariables(s); } 3125 3126 /// Breaks down a uri string to its components 3127 this(string uri) { 3128 reparse(uri); 3129 } 3130 3131 private void reparse(string uri) { 3132 // from RFC 3986 3133 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3134 // it was a nice experiment but just not worth it. 3135 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3136 /* 3137 Captures: 3138 0 = whole url 3139 1 = scheme, with : 3140 2 = scheme, no : 3141 3 = authority, with // 3142 4 = authority, no // 3143 5 = path 3144 6 = query string, with ? 3145 7 = query string, no ? 3146 8 = anchor, with # 3147 9 = anchor, no # 3148 */ 3149 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3150 // instead, I will DIY and cut that down to 0.6s on the same computer. 3151 /* 3152 3153 Note that authority is 3154 user:password@domain:port 3155 where the user:password@ part is optional, and the :port is optional. 3156 3157 Regex translation: 3158 3159 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3160 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3161 Path cannot have any ? or # in it. It is optional. 3162 Query must start with ? and must not have # in it. It is optional. 3163 Anchor must start with # and can have anything else in it to end of string. It is optional. 3164 */ 3165 3166 this = Uri.init; // reset all state 3167 3168 // empty uri = nothing special 3169 if(uri.length == 0) { 3170 return; 3171 } 3172 3173 size_t idx; 3174 3175 scheme_loop: foreach(char c; uri[idx .. $]) { 3176 switch(c) { 3177 case ':': 3178 case '/': 3179 case '?': 3180 case '#': 3181 break scheme_loop; 3182 default: 3183 } 3184 idx++; 3185 } 3186 3187 if(idx == 0 && uri[idx] == ':') { 3188 // this is actually a path! we skip way ahead 3189 goto path_loop; 3190 } 3191 3192 if(idx == uri.length) { 3193 // the whole thing is a path, apparently 3194 path = uri; 3195 return; 3196 } 3197 3198 if(idx > 0 && uri[idx] == ':') { 3199 scheme = uri[0 .. idx]; 3200 idx++; 3201 } else { 3202 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3203 idx = 0; 3204 } 3205 3206 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3207 // we have an authority.... 3208 idx += 2; 3209 3210 auto authority_start = idx; 3211 authority_loop: foreach(char c; uri[idx .. $]) { 3212 switch(c) { 3213 case '/': 3214 case '?': 3215 case '#': 3216 break authority_loop; 3217 default: 3218 } 3219 idx++; 3220 } 3221 3222 auto authority = uri[authority_start .. idx]; 3223 3224 auto idx2 = authority.indexOf("@"); 3225 if(idx2 != -1) { 3226 userinfo = authority[0 .. idx2]; 3227 authority = authority[idx2 + 1 .. $]; 3228 } 3229 3230 if(authority.length && authority[0] == '[') { 3231 // ipv6 address special casing 3232 idx2 = authority.indexOf(']'); 3233 if(idx2 != -1) { 3234 auto end = authority[idx2 + 1 .. $]; 3235 if(end.length && end[0] == ':') 3236 idx2 = idx2 + 1; 3237 else 3238 idx2 = -1; 3239 } 3240 } else { 3241 idx2 = authority.indexOf(":"); 3242 } 3243 3244 if(idx2 == -1) { 3245 port = 0; // 0 means not specified; we should use the default for the scheme 3246 host = authority; 3247 } else { 3248 host = authority[0 .. idx2]; 3249 if(idx2 + 1 < authority.length) 3250 port = to!int(authority[idx2 + 1 .. $]); 3251 else 3252 port = 0; 3253 } 3254 } 3255 3256 path_loop: 3257 auto path_start = idx; 3258 3259 foreach(char c; uri[idx .. $]) { 3260 if(c == '?' || c == '#') 3261 break; 3262 idx++; 3263 } 3264 3265 path = uri[path_start .. idx]; 3266 3267 if(idx == uri.length) 3268 return; // nothing more to examine... 3269 3270 if(uri[idx] == '?') { 3271 idx++; 3272 auto query_start = idx; 3273 foreach(char c; uri[idx .. $]) { 3274 if(c == '#') 3275 break; 3276 idx++; 3277 } 3278 query = uri[query_start .. idx]; 3279 } 3280 3281 if(idx < uri.length && uri[idx] == '#') { 3282 idx++; 3283 fragment = uri[idx .. $]; 3284 } 3285 3286 // uriInvalidated = false; 3287 } 3288 3289 private string rebuildUri() const { 3290 string ret; 3291 if(scheme.length) 3292 ret ~= scheme ~ ":"; 3293 if(userinfo.length || host.length) 3294 ret ~= "//"; 3295 if(userinfo.length) 3296 ret ~= userinfo ~ "@"; 3297 if(host.length) 3298 ret ~= host; 3299 if(port) 3300 ret ~= ":" ~ to!string(port); 3301 3302 ret ~= path; 3303 3304 if(query.length) 3305 ret ~= "?" ~ query; 3306 3307 if(fragment.length) 3308 ret ~= "#" ~ fragment; 3309 3310 // uri = ret; 3311 // uriInvalidated = false; 3312 return ret; 3313 } 3314 3315 /// Converts the broken down parts back into a complete string 3316 string toString() const { 3317 // if(uriInvalidated) 3318 return rebuildUri(); 3319 } 3320 3321 /// Returns a new absolute Uri given a base. It treats this one as 3322 /// relative where possible, but absolute if not. (If protocol, domain, or 3323 /// other info is not set, the new one inherits it from the base.) 3324 /// 3325 /// Browsers use a function like this to figure out links in html. 3326 Uri basedOn(in Uri baseUrl) const { 3327 Uri n = this; // copies 3328 if(n.scheme == "data") 3329 return n; 3330 // n.uriInvalidated = true; // make sure we regenerate... 3331 3332 // userinfo is not inherited... is this wrong? 3333 3334 // if anything is given in the existing url, we don't use the base anymore. 3335 if(n.scheme.empty) { 3336 n.scheme = baseUrl.scheme; 3337 if(n.host.empty) { 3338 n.host = baseUrl.host; 3339 if(n.port == 0) { 3340 n.port = baseUrl.port; 3341 if(n.path.length > 0 && n.path[0] != '/') { 3342 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3343 if(b.length == 0) 3344 b = "/"; 3345 n.path = b ~ n.path; 3346 } else if(n.path.length == 0) { 3347 n.path = baseUrl.path; 3348 } 3349 } 3350 } 3351 } 3352 3353 n.removeDots(); 3354 3355 return n; 3356 } 3357 3358 void removeDots() { 3359 auto parts = this.path.split("/"); 3360 string[] toKeep; 3361 foreach(part; parts) { 3362 if(part == ".") { 3363 continue; 3364 } else if(part == "..") { 3365 //if(toKeep.length > 1) 3366 toKeep = toKeep[0 .. $-1]; 3367 //else 3368 //toKeep = [""]; 3369 continue; 3370 } else { 3371 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3372 //continue; // skip a `//` situation 3373 toKeep ~= part; 3374 } 3375 } 3376 3377 auto path = toKeep.join("/"); 3378 if(path.length && path[0] != '/') 3379 path = "/" ~ path; 3380 3381 this.path = path; 3382 } 3383 3384 unittest { 3385 auto uri = Uri("test.html"); 3386 assert(uri.path == "test.html"); 3387 uri = Uri("path/1/lol"); 3388 assert(uri.path == "path/1/lol"); 3389 uri = Uri("http://me@example.com"); 3390 assert(uri.scheme == "http"); 3391 assert(uri.userinfo == "me"); 3392 assert(uri.host == "example.com"); 3393 uri = Uri("http://example.com/#a"); 3394 assert(uri.scheme == "http"); 3395 assert(uri.host == "example.com"); 3396 assert(uri.fragment == "a"); 3397 uri = Uri("#foo"); 3398 assert(uri.fragment == "foo"); 3399 uri = Uri("?lol"); 3400 assert(uri.query == "lol"); 3401 uri = Uri("#foo?lol"); 3402 assert(uri.fragment == "foo?lol"); 3403 uri = Uri("?lol#foo"); 3404 assert(uri.fragment == "foo"); 3405 assert(uri.query == "lol"); 3406 3407 uri = Uri("http://127.0.0.1/"); 3408 assert(uri.host == "127.0.0.1"); 3409 assert(uri.port == 0); 3410 3411 uri = Uri("http://127.0.0.1:123/"); 3412 assert(uri.host == "127.0.0.1"); 3413 assert(uri.port == 123); 3414 3415 uri = Uri("http://[ff:ff::0]/"); 3416 assert(uri.host == "[ff:ff::0]"); 3417 3418 uri = Uri("http://[ff:ff::0]:123/"); 3419 assert(uri.host == "[ff:ff::0]"); 3420 assert(uri.port == 123); 3421 } 3422 3423 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3424 // the possibilities. 3425 unittest { 3426 auto url = Uri("cool.html"); // checking relative links 3427 3428 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3429 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3430 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3431 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3432 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3433 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3434 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3435 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3436 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3437 3438 url = Uri("/something/cool.html"); // same server, different path 3439 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3440 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3441 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3442 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3443 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3444 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3445 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3446 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3447 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3448 3449 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3450 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3451 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3452 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3453 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3454 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3455 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3456 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3457 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3458 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3459 3460 url = Uri("/test/bar"); 3461 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3462 assert(Uri("../").basedOn(url) == "/"); 3463 3464 url = Uri("http://example.com/"); 3465 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3466 3467 //auto uriBefore = url; 3468 url = Uri("#anchor"); // everything should remain the same except the anchor 3469 //uriBefore.anchor = "anchor"); 3470 //assert(url == uriBefore); 3471 3472 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3473 3474 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3475 3476 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3477 3478 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3479 3480 // FIXME: add something for port too 3481 } 3482 3483 // these are like javascript's location.search and location.hash 3484 string search() const { 3485 return query.length ? ("?" ~ query) : ""; 3486 } 3487 string hash() const { 3488 return fragment.length ? ("#" ~ fragment) : ""; 3489 } 3490 } 3491 3492 3493 /* 3494 for session, see web.d 3495 */ 3496 3497 /// breaks down a url encoded string 3498 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3499 auto vars = data.split(separator); 3500 string[][string] _get; 3501 foreach(var; vars) { 3502 auto equal = var.indexOf("="); 3503 string name; 3504 string value; 3505 if(equal == -1) { 3506 name = decodeComponent(var); 3507 value = ""; 3508 } else { 3509 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3510 // stupid + -> space conversion. 3511 name = decodeComponent(var[0..equal].replace("+", " ")); 3512 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3513 } 3514 3515 _get[name] ~= value; 3516 if(namesInOrder) 3517 (*namesInOrder) ~= name; 3518 if(valuesInOrder) 3519 (*valuesInOrder) ~= value; 3520 } 3521 return _get; 3522 } 3523 3524 /// breaks down a url encoded string, but only returns the last value of any array 3525 string[string] decodeVariablesSingle(string data) { 3526 string[string] va; 3527 auto varArray = decodeVariables(data); 3528 foreach(k, v; varArray) 3529 va[k] = v[$-1]; 3530 3531 return va; 3532 } 3533 3534 /// url encodes the whole string 3535 string encodeVariables(in string[string] data) { 3536 string ret; 3537 3538 bool outputted = false; 3539 foreach(k, v; data) { 3540 if(outputted) 3541 ret ~= "&"; 3542 else 3543 outputted = true; 3544 3545 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3546 } 3547 3548 return ret; 3549 } 3550 3551 /// url encodes a whole string 3552 string encodeVariables(in string[][string] data) { 3553 string ret; 3554 3555 bool outputted = false; 3556 foreach(k, arr; data) { 3557 foreach(v; arr) { 3558 if(outputted) 3559 ret ~= "&"; 3560 else 3561 outputted = true; 3562 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3563 } 3564 } 3565 3566 return ret; 3567 } 3568 3569 /// Encodes all but the explicitly unreserved characters per rfc 3986 3570 /// Alphanumeric and -_.~ are the only ones left unencoded 3571 /// name is borrowed from php 3572 string rawurlencode(in char[] data) { 3573 string ret; 3574 ret.reserve(data.length * 2); 3575 foreach(char c; data) { 3576 if( 3577 (c >= 'a' && c <= 'z') || 3578 (c >= 'A' && c <= 'Z') || 3579 (c >= '0' && c <= '9') || 3580 c == '-' || c == '_' || c == '.' || c == '~') 3581 { 3582 ret ~= c; 3583 } else { 3584 ret ~= '%'; 3585 // since we iterate on char, this should give us the octets of the full utf8 string 3586 ret ~= toHexUpper(c); 3587 } 3588 } 3589 3590 return ret; 3591 } 3592 3593 3594 // http helper functions 3595 3596 // for chunked responses (which embedded http does whenever possible) 3597 version(none) // this is moved up above to avoid making a copy of the data 3598 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3599 const(ubyte)[] ret; 3600 3601 ret = cast(const(ubyte)[]) toHex(data.length); 3602 ret ~= cast(const(ubyte)[]) "\r\n"; 3603 ret ~= data; 3604 ret ~= cast(const(ubyte)[]) "\r\n"; 3605 3606 return ret; 3607 } 3608 3609 string toHex(long num) { 3610 string ret; 3611 while(num) { 3612 int v = num % 16; 3613 num /= 16; 3614 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3615 ret ~= d; 3616 } 3617 3618 return to!string(array(ret.retro)); 3619 } 3620 3621 string toHexUpper(long num) { 3622 string ret; 3623 while(num) { 3624 int v = num % 16; 3625 num /= 16; 3626 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3627 ret ~= d; 3628 } 3629 3630 if(ret.length == 1) 3631 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3632 3633 return to!string(array(ret.retro)); 3634 } 3635 3636 3637 // the generic mixins 3638 3639 /++ 3640 Use this instead of writing your own main 3641 3642 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3643 +/ 3644 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3645 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3646 } 3647 3648 /++ 3649 Boilerplate mixin for a main function that uses the [dispatcher] function. 3650 3651 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3652 3653 History: 3654 Added July 9, 2021 3655 +/ 3656 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3657 /// forwards to [CustomCgiDispatcherMain] with default args 3658 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3659 } 3660 3661 /// ditto 3662 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3663 class GenericPresenter : WebPresenter!GenericPresenter {} 3664 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3665 } 3666 3667 /++ 3668 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3669 3670 History: 3671 Added May 13, 2023 (dub v11.0) 3672 +/ 3673 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3674 /++ 3675 Handler to the generated presenter you can use from your objects, etc. 3676 +/ 3677 Presenter activePresenter; 3678 3679 /++ 3680 Request handler that creates the presenter then forwards to the [dispatcher] function. 3681 Renders 404 if the dispatcher did not handle the request. 3682 3683 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3684 +/ 3685 void handler(Cgi cgi) { 3686 auto presenter = new Presenter; 3687 activePresenter = presenter; 3688 scope(exit) activePresenter = null; 3689 3690 if(cgi.dispatcher!DispatcherArgs(presenter)) 3691 return; 3692 3693 switch(cgi.pathInfo) { 3694 case "/style.css": 3695 cgi.setCache(true); 3696 cgi.setResponseContentType("text/css"); 3697 cgi.write(presenter.style(), true); 3698 break; 3699 case "/script.js": 3700 cgi.setCache(true); 3701 cgi.setResponseContentType("application/javascript"); 3702 cgi.write(presenter.script(), true); 3703 break; 3704 default: 3705 presenter.renderBasicError(cgi, 404); 3706 } 3707 } 3708 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3709 } 3710 3711 /// ditto 3712 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3713 class GenericPresenter : WebPresenter!GenericPresenter {} 3714 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3715 3716 } 3717 3718 private string simpleHtmlEncode(string s) { 3719 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3720 } 3721 3722 string messageFromException(Throwable t) { 3723 string message; 3724 if(t !is null) { 3725 debug message = t.toString(); 3726 else message = "An unexpected error has occurred."; 3727 } else { 3728 message = "Unknown error"; 3729 } 3730 return message; 3731 } 3732 3733 string plainHttpError(bool isCgi, string type, Throwable t) { 3734 auto message = messageFromException(t); 3735 message = simpleHtmlEncode(message); 3736 3737 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3738 isCgi ? "Status:" : "HTTP/1.1", 3739 type, message.length, message); 3740 } 3741 3742 // returns true if we were able to recover reasonably 3743 bool handleException(Cgi cgi, Throwable t) { 3744 if(cgi.isClosed) { 3745 // if the channel has been explicitly closed, we can't handle it here 3746 return true; 3747 } 3748 3749 if(cgi.outputtedResponseData) { 3750 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3751 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3752 } else { 3753 // no headers are sent, we can send a full blown error and recover 3754 cgi.setCache(false); 3755 cgi.setResponseContentType("text/html"); 3756 cgi.setResponseLocation(null); // cancel the redirect 3757 cgi.setResponseStatus("500 Internal Server Error"); 3758 cgi.write(simpleHtmlEncode(messageFromException(t))); 3759 cgi.close(); 3760 return true; 3761 } 3762 } 3763 3764 bool isCgiRequestMethod(string s) { 3765 s = s.toUpper(); 3766 if(s == "COMMANDLINE") 3767 return true; 3768 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3769 if(s == member) 3770 return true; 3771 return false; 3772 } 3773 3774 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3775 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3776 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3777 void main(string[] args) { 3778 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3779 } 3780 } 3781 3782 version(embedded_httpd_processes) 3783 __gshared int processPoolSize = 8; 3784 3785 // Returns true if run. You should exit the program after that. 3786 bool tryAddonServers(string[] args) { 3787 if(args.length > 1) { 3788 // run the special separate processes if needed 3789 switch(args[1]) { 3790 case "--websocket-server": 3791 version(with_addon_servers) 3792 websocketServers[args[2]](args[3 .. $]); 3793 else 3794 printf("Add-on servers not compiled in.\n"); 3795 return true; 3796 case "--websocket-servers": 3797 import core.demangle; 3798 version(with_addon_servers_connections) 3799 foreach(k, v; websocketServers) 3800 writeln(k, "\t", demangle(k)); 3801 return true; 3802 case "--session-server": 3803 version(with_addon_servers) 3804 runSessionServer(); 3805 else 3806 printf("Add-on servers not compiled in.\n"); 3807 return true; 3808 case "--event-server": 3809 version(with_addon_servers) 3810 runEventServer(); 3811 else 3812 printf("Add-on servers not compiled in.\n"); 3813 return true; 3814 case "--timer-server": 3815 version(with_addon_servers) 3816 runTimerServer(); 3817 else 3818 printf("Add-on servers not compiled in.\n"); 3819 return true; 3820 case "--timed-jobs": 3821 import core.demangle; 3822 version(with_addon_servers_connections) 3823 foreach(k, v; scheduledJobHandlers) 3824 writeln(k, "\t", demangle(k)); 3825 return true; 3826 case "--timed-job": 3827 scheduledJobHandlers[args[2]](args[3 .. $]); 3828 return true; 3829 default: 3830 // intentionally blank - do nothing and carry on to run normally 3831 } 3832 } 3833 return false; 3834 } 3835 3836 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3837 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3838 // we support command line thing for easy testing everywhere 3839 // it needs to be called ./app method uri [other args...] 3840 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3841 Cgi cgi = new CustomCgi(args); 3842 scope(exit) cgi.dispose(); 3843 try { 3844 fun(cgi); 3845 cgi.close(); 3846 } catch(AuthorizationRequiredException are) { 3847 cgi.setResponseStatus("401 Authorization Required"); 3848 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 3849 cgi.close(); 3850 } 3851 writeln(); // just to put a blank line before the prompt cuz it annoys me 3852 // FIXME: put in some footers to show what changes happened in the session 3853 // could make the MockSession be some kind of ReflectableSessionObject or something 3854 return true; 3855 } 3856 return false; 3857 } 3858 3859 /++ 3860 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3861 3862 As of version 11 (released August 2023), you can also make things like this: 3863 3864 --- 3865 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3866 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3867 3868 // can also: 3869 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3870 3871 // NOT IMPLEMENTED YET 3872 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3873 3874 foreach(listenSpec; server.listenSpecs) { 3875 // you can check what it actually bound to here and see your assigned ports 3876 } 3877 3878 // NOT IMPLEMENTED YET 3879 // server.start!handler(); // starts and runs in the arsd.core event loop 3880 3881 server.serve!handler(); // blocks the thread until the server exits 3882 --- 3883 3884 History: 3885 Added Sept 26, 2020 (release version 8.5). 3886 3887 The `listenSpec` member was added July 31, 2023. 3888 +/ 3889 struct RequestServer { 3890 /++ 3891 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3892 +/ 3893 string listeningHost = defaultListeningHost(); 3894 /// ditto 3895 ushort listeningPort = defaultListeningPort(); 3896 3897 static struct ListenSpec { 3898 enum Protocol { 3899 http, 3900 https, 3901 scgi 3902 } 3903 Protocol protocol; 3904 3905 enum AddressType { 3906 ip, 3907 unix, 3908 abstract_ 3909 } 3910 AddressType addressType; 3911 3912 string address; 3913 ushort port; 3914 } 3915 3916 /++ 3917 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3918 3919 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3920 3921 `http://localhost:8080` 3922 3923 `http://unix:filename/here` 3924 3925 `scgi://abstract:/name/here` 3926 3927 `http://[::1]:4444` 3928 3929 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3930 3931 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3932 3933 `localhost:8080` serves the default protocol. 3934 3935 `8080` or `:8080` assumes default protocol on localhost. 3936 3937 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3938 3939 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3940 3941 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3942 3943 $(PITFALL 3944 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3945 ) 3946 3947 Bugs: 3948 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3949 3950 History: 3951 Added July 31, 2023 (dub v11.0) 3952 +/ 3953 string[] listenSpec; 3954 3955 /++ 3956 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3957 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3958 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3959 [stop] may not work as well. 3960 3961 History: 3962 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3963 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3964 compatibility. 3965 +/ 3966 bool useFork = cgi_use_fork_default; 3967 3968 /++ 3969 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3970 default based on the number of cpus modified by the server mode. 3971 3972 History: 3973 Added August 12, 2022 (dub v10.9) 3974 +/ 3975 int numberOfThreads = 0; 3976 3977 /++ 3978 Creates a server configured to listen to multiple URLs. 3979 3980 History: 3981 Added July 31, 2023 (dub v11.0) 3982 +/ 3983 this(string[] listenTo) { 3984 this.listenSpec = listenTo; 3985 } 3986 3987 /// Creates a server object configured to listen on a single host and port. 3988 this(string defaultHost, ushort defaultPort) { 3989 this.listeningHost = defaultHost; 3990 this.listeningPort = defaultPort; 3991 } 3992 3993 /// ditto 3994 this(ushort defaultPort) { 3995 listeningPort = defaultPort; 3996 } 3997 3998 /++ 3999 Reads the command line arguments into the values here. 4000 4001 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 4002 4003 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 4004 +/ 4005 void configureFromCommandLine(string[] args) { 4006 bool portOrHostFound = false; 4007 4008 bool foundPort = false; 4009 bool foundHost = false; 4010 bool foundUid = false; 4011 bool foundGid = false; 4012 bool foundListen = false; 4013 foreach(arg; args) { 4014 if(foundPort) { 4015 listeningPort = to!ushort(arg); 4016 portOrHostFound = true; 4017 foundPort = false; 4018 continue; 4019 } 4020 if(foundHost) { 4021 listeningHost = arg; 4022 portOrHostFound = true; 4023 foundHost = false; 4024 continue; 4025 } 4026 if(foundUid) { 4027 privilegesDropToUid = to!uid_t(arg); 4028 foundUid = false; 4029 continue; 4030 } 4031 if(foundGid) { 4032 privilegesDropToGid = to!gid_t(arg); 4033 foundGid = false; 4034 continue; 4035 } 4036 if(foundListen) { 4037 this.listenSpec ~= arg; 4038 foundListen = false; 4039 continue; 4040 } 4041 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 4042 foundHost = true; 4043 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 4044 foundPort = true; 4045 else if(arg == "--uid") 4046 foundUid = true; 4047 else if(arg == "--gid") 4048 foundGid = true; 4049 else if(arg == "--listen") 4050 foundListen = true; 4051 } 4052 4053 if(portOrHostFound && listenSpec.length) { 4054 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 4055 } 4056 } 4057 4058 version(Windows) { 4059 private alias uid_t = int; 4060 private alias gid_t = int; 4061 } 4062 4063 /// user (uid) to drop privileges to 4064 /// 0 … do nothing 4065 uid_t privilegesDropToUid = 0; 4066 /// group (gid) to drop privileges to 4067 /// 0 … do nothing 4068 gid_t privilegesDropToGid = 0; 4069 4070 private void dropPrivileges() { 4071 version(Posix) { 4072 import core.sys.posix.unistd; 4073 4074 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 4075 throw new Exception("Dropping privileges via setgid() failed."); 4076 4077 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 4078 throw new Exception("Dropping privileges via setuid() failed."); 4079 } 4080 else { 4081 // FIXME: Windows? 4082 //pragma(msg, "Dropping privileges is not implemented for this platform"); 4083 } 4084 4085 // done, set zero 4086 privilegesDropToGid = 0; 4087 privilegesDropToUid = 0; 4088 } 4089 4090 /++ 4091 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 4092 4093 History: 4094 Added Oct 10, 2020. 4095 Example: 4096 4097 --- 4098 import arsd.cgi; 4099 void main() { 4100 RequestServer server = RequestServer("127.0.0.1", 6789); 4101 string oauthCode; 4102 string oauthScope; 4103 server.serveHttpOnce!((cgi) { 4104 oauthCode = cgi.request("code"); 4105 oauthScope = cgi.request("scope"); 4106 cgi.write("Thank you, please return to the application."); 4107 }); 4108 // use the code and scope given 4109 } 4110 --- 4111 +/ 4112 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4113 import std.socket; 4114 4115 bool tcp; 4116 void delegate() cleanup; 4117 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4118 auto connection = socket.accept(); 4119 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4120 4121 if(cleanup) 4122 cleanup(); 4123 } 4124 4125 /++ 4126 Starts serving requests according to the current configuration. 4127 +/ 4128 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4129 version(netman_httpd) { 4130 // Obsolete! 4131 4132 import arsd.httpd; 4133 // what about forwarding the other constructor args? 4134 // this probably needs a whole redoing... 4135 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4136 return; 4137 } else 4138 version(embedded_httpd_processes) { 4139 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4140 } else 4141 version(embedded_httpd_threads) { 4142 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4143 } else 4144 version(scgi) { 4145 serveScgi!(fun, CustomCgi, maxContentLength)(); 4146 } else 4147 version(fastcgi) { 4148 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4149 } else 4150 version(stdio_http) { 4151 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4152 } else { 4153 //version=plain_cgi; 4154 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4155 } 4156 } 4157 4158 /++ 4159 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4160 4161 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4162 +/ 4163 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4164 globalStopFlag = false; 4165 static if(__traits(isStaticFunction, fun)) 4166 alias funToUse = fun; 4167 else 4168 void funToUse(CustomCgi cgi) { 4169 static if(__VERSION__ > 2097) 4170 __traits(child, _this, fun)(cgi); 4171 else static assert(0, "Not implemented in your compiler version!"); 4172 } 4173 auto manager = this.listenSpec is null ? 4174 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4175 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4176 manager.listen(); 4177 } 4178 4179 /++ 4180 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4181 +/ 4182 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4183 globalStopFlag = false; 4184 auto manager = this.listenSpec is null ? 4185 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4186 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4187 manager.listen(); 4188 } 4189 4190 /++ 4191 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4192 4193 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4194 4195 History: 4196 Added May 29, 2021 4197 +/ 4198 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4199 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4200 } 4201 4202 /++ 4203 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4204 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4205 +/ 4206 enum ForceStop { 4207 /++ 4208 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4209 +/ 4210 afterQueuedRequestsComplete, 4211 /++ 4212 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4213 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4214 +/ 4215 afterCurrentRequestsComplete, 4216 /++ 4217 Partial response writes will throw an exception, cancelling any streaming response, but complete 4218 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4219 +/ 4220 cancelStreamingRequestsEarly, 4221 /++ 4222 All writes will throw. 4223 +/ 4224 cancelAllRequestsEarly, 4225 /++ 4226 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4227 +/ 4228 forciblyTerminate, 4229 } 4230 4231 version(embedded_httpd_processes) {} else 4232 /++ 4233 Stops serving after the current requests are completed. 4234 4235 Bugs: 4236 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4237 on Windows however). Only partially implemented on non-Linux posix systems. 4238 4239 You might also try SIGINT perhaps. 4240 4241 The stopPriority is not yet fully implemented. 4242 +/ 4243 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4244 globalStopFlag = true; 4245 4246 version(Posix) { 4247 if(cancelfd > 0) { 4248 ulong a = 1; 4249 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4250 } 4251 } 4252 version(Windows) { 4253 if(iocp) { 4254 foreach(i; 0 .. 16) // FIXME 4255 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4256 } 4257 } 4258 } 4259 } 4260 4261 class AuthorizationRequiredException : Exception { 4262 string type; 4263 string realm; 4264 this(string type, string realm, string file, size_t line) { 4265 this.type = type; 4266 this.realm = realm; 4267 4268 super("Authorization Required", file, line); 4269 } 4270 } 4271 4272 private alias AliasSeq(T...) = T; 4273 4274 version(with_breaking_cgi_features) 4275 mixin(q{ 4276 template ThisFor(alias t) { 4277 static if(__traits(isStaticFunction, t)) { 4278 alias ThisFor = AliasSeq!(); 4279 } else { 4280 alias ThisFor = __traits(parent, t); 4281 } 4282 } 4283 }); 4284 else 4285 alias ThisFor(alias t) = AliasSeq!(); 4286 4287 private __gshared bool globalStopFlag = false; 4288 4289 version(embedded_httpd_processes) 4290 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4291 import core.sys.posix.unistd; 4292 import core.sys.posix.sys.socket; 4293 import core.sys.posix.netinet.in_; 4294 //import std.c.linux.socket; 4295 4296 int sock = socket(AF_INET, SOCK_STREAM, 0); 4297 if(sock == -1) 4298 throw new Exception("socket"); 4299 4300 cloexec(sock); 4301 4302 { 4303 4304 sockaddr_in addr; 4305 addr.sin_family = AF_INET; 4306 addr.sin_port = htons(params.listeningPort); 4307 auto lh = params.listeningHost; 4308 if(lh.length) { 4309 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4310 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4311 } else 4312 addr.sin_addr.s_addr = INADDR_ANY; 4313 4314 // HACKISH 4315 int on = 1; 4316 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4317 // end hack 4318 4319 4320 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4321 close(sock); 4322 throw new Exception("bind"); 4323 } 4324 4325 // FIXME: if this queue is full, it will just ignore it 4326 // and wait for the client to retransmit it. This is an 4327 // obnoxious timeout condition there. 4328 if(sock.listen(128) == -1) { 4329 close(sock); 4330 throw new Exception("listen"); 4331 } 4332 params.dropPrivileges(); 4333 } 4334 4335 version(embedded_httpd_processes_accept_after_fork) {} else { 4336 int pipeReadFd; 4337 int pipeWriteFd; 4338 4339 { 4340 int[2] pipeFd; 4341 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4342 import core.stdc.errno; 4343 throw new Exception("pipe failed " ~ to!string(errno)); 4344 } 4345 4346 pipeReadFd = pipeFd[0]; 4347 pipeWriteFd = pipeFd[1]; 4348 } 4349 } 4350 4351 4352 int processCount; 4353 pid_t newPid; 4354 reopen: 4355 while(processCount < processPoolSize) { 4356 newPid = fork(); 4357 if(newPid == 0) { 4358 // start serving on the socket 4359 //ubyte[4096] backingBuffer; 4360 for(;;) { 4361 bool closeConnection; 4362 uint i; 4363 sockaddr addr; 4364 i = addr.sizeof; 4365 version(embedded_httpd_processes_accept_after_fork) { 4366 int s = accept(sock, &addr, &i); 4367 int opt = 1; 4368 import core.sys.posix.netinet.tcp; 4369 // the Cgi class does internal buffering, so disabling this 4370 // helps with latency in many cases... 4371 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4372 cloexec(s); 4373 } else { 4374 int s; 4375 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4376 if(readret != s.sizeof) { 4377 import core.stdc.errno; 4378 throw new Exception("pipe read failed " ~ to!string(errno)); 4379 } 4380 4381 //writeln("process ", getpid(), " got socket ", s); 4382 } 4383 4384 try { 4385 4386 if(s == -1) 4387 throw new Exception("accept"); 4388 4389 scope(failure) close(s); 4390 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4391 auto ir = new BufferedInputRange(s); 4392 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4393 4394 while(!ir.empty) { 4395 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4396 4397 Cgi cgi; 4398 try { 4399 cgi = new CustomCgi(ir, &closeConnection); 4400 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4401 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4402 if(processPoolSize <= 1) 4403 closeConnection = true; 4404 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4405 } catch(HttpVersionNotSupportedException he) { 4406 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4407 closeConnection = true; 4408 break; 4409 } catch(Throwable t) { 4410 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4411 // anyway let's kill the connection 4412 version(CRuntime_Musl) { 4413 // LockingTextWriter fails here 4414 // so working around it 4415 auto estr = t.toString(); 4416 stderr.rawWrite(estr); 4417 stderr.rawWrite("\n"); 4418 } else 4419 stderr.writeln(t.toString()); 4420 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4421 closeConnection = true; 4422 break; 4423 } 4424 assert(cgi !is null); 4425 scope(exit) 4426 cgi.dispose(); 4427 4428 try { 4429 fun(cgi); 4430 cgi.close(); 4431 if(cgi.websocketMode) 4432 closeConnection = true; 4433 4434 } catch(AuthorizationRequiredException are) { 4435 cgi.setResponseStatus("401 Authorization Required"); 4436 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4437 cgi.close(); 4438 } catch(ConnectionException ce) { 4439 closeConnection = true; 4440 } catch(Throwable t) { 4441 // a processing error can be recovered from 4442 version(CRuntime_Musl) { 4443 // LockingTextWriter fails here 4444 // so working around it 4445 auto estr = t.toString(); 4446 stderr.rawWrite(estr); 4447 } else { 4448 stderr.writeln(t.toString); 4449 } 4450 if(!handleException(cgi, t)) 4451 closeConnection = true; 4452 } 4453 4454 if(closeConnection) { 4455 ir.source.close(); 4456 break; 4457 } else { 4458 if(!ir.empty) 4459 ir.popFront(); // get the next 4460 else if(ir.sourceClosed) { 4461 ir.source.close(); 4462 } 4463 } 4464 } 4465 4466 ir.source.close(); 4467 } catch(Throwable t) { 4468 version(CRuntime_Musl) {} else 4469 debug writeln(t); 4470 // most likely cause is a timeout 4471 } 4472 } 4473 } else if(newPid < 0) { 4474 throw new Exception("fork failed"); 4475 } else { 4476 processCount++; 4477 } 4478 } 4479 4480 // the parent should wait for its children... 4481 if(newPid) { 4482 import core.sys.posix.sys.wait; 4483 4484 version(embedded_httpd_processes_accept_after_fork) {} else { 4485 import core.sys.posix.sys.select; 4486 int[] fdQueue; 4487 while(true) { 4488 // writeln("select call"); 4489 int nfds = pipeWriteFd; 4490 if(sock > pipeWriteFd) 4491 nfds = sock; 4492 nfds += 1; 4493 fd_set read_fds; 4494 fd_set write_fds; 4495 FD_ZERO(&read_fds); 4496 FD_ZERO(&write_fds); 4497 FD_SET(sock, &read_fds); 4498 if(fdQueue.length) 4499 FD_SET(pipeWriteFd, &write_fds); 4500 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4501 if(ret == -1) { 4502 import core.stdc.errno; 4503 if(errno == EINTR) 4504 goto try_wait; 4505 else 4506 throw new Exception("wtf select"); 4507 } 4508 4509 int s = -1; 4510 if(FD_ISSET(sock, &read_fds)) { 4511 uint i; 4512 sockaddr addr; 4513 i = addr.sizeof; 4514 s = accept(sock, &addr, &i); 4515 cloexec(s); 4516 import core.sys.posix.netinet.tcp; 4517 int opt = 1; 4518 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4519 } 4520 4521 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4522 if(s == -1 && fdQueue.length) { 4523 s = fdQueue[0]; 4524 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4525 } 4526 write_fd(pipeWriteFd, &s, s.sizeof, s); 4527 close(s); // we are done with it, let the other process take ownership 4528 } else 4529 fdQueue ~= s; 4530 } 4531 } 4532 4533 try_wait: 4534 4535 int status; 4536 while(-1 != wait(&status)) { 4537 version(CRuntime_Musl) {} else { 4538 import std.stdio; writeln("Process died ", status); 4539 } 4540 processCount--; 4541 goto reopen; 4542 } 4543 close(sock); 4544 } 4545 } 4546 4547 version(fastcgi) 4548 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4549 // SetHandler fcgid-script 4550 FCGX_Stream* input, output, error; 4551 FCGX_ParamArray env; 4552 4553 4554 4555 const(ubyte)[] getFcgiChunk() { 4556 const(ubyte)[] ret; 4557 while(FCGX_HasSeenEOF(input) != -1) 4558 ret ~= cast(ubyte) FCGX_GetChar(input); 4559 return ret; 4560 } 4561 4562 void writeFcgi(const(ubyte)[] data) { 4563 FCGX_PutStr(data.ptr, data.length, output); 4564 } 4565 4566 void doARequest() { 4567 string[string] fcgienv; 4568 4569 for(auto e = env; e !is null && *e !is null; e++) { 4570 string cur = to!string(*e); 4571 auto idx = cur.indexOf("="); 4572 string name, value; 4573 if(idx == -1) 4574 name = cur; 4575 else { 4576 name = cur[0 .. idx]; 4577 value = cur[idx + 1 .. $]; 4578 } 4579 4580 fcgienv[name] = value; 4581 } 4582 4583 void flushFcgi() { 4584 FCGX_FFlush(output); 4585 } 4586 4587 Cgi cgi; 4588 try { 4589 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4590 } catch(Throwable t) { 4591 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4592 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4593 return; //continue; 4594 } 4595 assert(cgi !is null); 4596 scope(exit) cgi.dispose(); 4597 try { 4598 fun(cgi); 4599 cgi.close(); 4600 } catch(AuthorizationRequiredException are) { 4601 cgi.setResponseStatus("401 Authorization Required"); 4602 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4603 cgi.close(); 4604 } catch(Throwable t) { 4605 // log it to the error stream 4606 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4607 // handle it for the user, if we can 4608 if(!handleException(cgi, t)) 4609 return; // continue; 4610 } 4611 } 4612 4613 auto lp = params.listeningPort; 4614 auto host = params.listeningHost; 4615 4616 FCGX_Request request; 4617 if(lp || !host.empty) { 4618 // if a listening port was specified on the command line, we want to spawn ourself 4619 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4620 FCGX_Init(); 4621 4622 int sock; 4623 4624 if(host.startsWith("unix:")) { 4625 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4626 } else if(host.startsWith("abstract:")) { 4627 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4628 } else { 4629 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4630 } 4631 4632 if(sock < 0) 4633 throw new Exception("Couldn't listen on the port"); 4634 FCGX_InitRequest(&request, sock, 0); 4635 while(FCGX_Accept_r(&request) >= 0) { 4636 input = request.inStream; 4637 output = request.outStream; 4638 error = request.errStream; 4639 env = request.envp; 4640 doARequest(); 4641 } 4642 } else { 4643 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4644 // using the version with a global variable since we are separate processes anyway 4645 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4646 doARequest(); 4647 } 4648 } 4649 } 4650 4651 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4652 ushort defaultListeningPort() { 4653 version(netman_httpd) 4654 return 8080; 4655 else version(embedded_httpd_processes) 4656 return 8085; 4657 else version(embedded_httpd_threads) 4658 return 8085; 4659 else version(scgi) 4660 return 4000; 4661 else 4662 return 0; 4663 } 4664 4665 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4666 string defaultListeningHost() { 4667 version(netman_httpd) 4668 return null; 4669 else version(embedded_httpd_processes) 4670 return null; 4671 else version(embedded_httpd_threads) 4672 return null; 4673 else version(scgi) 4674 return "127.0.0.1"; 4675 else 4676 return null; 4677 4678 } 4679 4680 /++ 4681 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4682 4683 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4684 4685 Params: 4686 fun = Your request handler 4687 CustomCgi = a subclass of Cgi, if you wise to customize it further 4688 maxContentLength = max POST size you want to allow 4689 args = command-line arguments 4690 4691 History: 4692 Documented Sept 26, 2020. 4693 +/ 4694 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4695 if(tryAddonServers(args)) 4696 return; 4697 4698 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4699 return; 4700 4701 RequestServer server; 4702 // you can change the port here if you like 4703 // server.listeningPort = 9000; 4704 4705 // then call this to let the command line args override your default 4706 server.configureFromCommandLine(args); 4707 4708 // and serve the request(s). 4709 server.serve!(fun, CustomCgi, maxContentLength)(); 4710 } 4711 4712 //version(plain_cgi) 4713 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4714 // standard CGI is the default version 4715 4716 4717 // Set stdin to binary mode if necessary to avoid mangled newlines 4718 // the fact that stdin is global means this could be trouble but standard cgi request 4719 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4720 version(Windows) { 4721 version(Win64) 4722 _setmode(std.stdio.stdin.fileno(), 0x8000); 4723 else 4724 setmode(std.stdio.stdin.fileno(), 0x8000); 4725 } 4726 4727 Cgi cgi; 4728 try { 4729 cgi = new CustomCgi(maxContentLength); 4730 version(Posix) 4731 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4732 else version(Windows) 4733 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4734 else static assert(0); 4735 } catch(Throwable t) { 4736 version(CRuntime_Musl) { 4737 // LockingTextWriter fails here 4738 // so working around it 4739 auto s = t.toString(); 4740 stderr.rawWrite(s); 4741 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4742 } else { 4743 stderr.writeln(t.msg); 4744 // the real http server will probably handle this; 4745 // most likely, this is a bug in Cgi. But, oh well. 4746 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4747 } 4748 return; 4749 } 4750 assert(cgi !is null); 4751 scope(exit) cgi.dispose(); 4752 4753 try { 4754 fun(cgi); 4755 cgi.close(); 4756 } catch(AuthorizationRequiredException are) { 4757 cgi.setResponseStatus("401 Authorization Required"); 4758 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 4759 cgi.close(); 4760 } catch (Throwable t) { 4761 version(CRuntime_Musl) { 4762 // LockingTextWriter fails here 4763 // so working around it 4764 auto s = t.msg; 4765 stderr.rawWrite(s); 4766 } else { 4767 stderr.writeln(t.msg); 4768 } 4769 if(!handleException(cgi, t)) 4770 return; 4771 } 4772 } 4773 4774 private __gshared int cancelfd = -1; 4775 4776 /+ 4777 The event loop for embedded_httpd_threads will prolly fiber dispatch 4778 cgi constructors too, so slow posts will not monopolize a worker thread. 4779 4780 May want to provide the worker task system just need to ensure all the fibers 4781 has a big enough stack for real work... would also ideally like to reuse them. 4782 4783 4784 So prolly bir would switch it to nonblocking. If it would block, it epoll 4785 registers one shot with this existing fiber to take it over. 4786 4787 new connection comes in. it picks a fiber off the free list, 4788 or if there is none, it creates a new one. this fiber handles 4789 this connection the whole time. 4790 4791 epoll triggers the fiber when something comes in. it is called by 4792 a random worker thread, it might change at any time. at least during 4793 the constructor. maybe into the main body it will stay tied to a thread 4794 just so TLS stuff doesn't randomly change in the middle. but I could 4795 specify if you yield all bets are off. 4796 4797 when the request is finished, if there's more data buffered, it just 4798 keeps going. if there is no more data buffered, it epoll ctls to 4799 get triggered when more data comes in. all one shot. 4800 4801 when a connection is closed, the fiber returns and is then reset 4802 and added to the free list. if the free list is full, the fiber is 4803 just freed, this means it will balloon to a certain size but not generally 4804 grow beyond that unless the activity keeps going. 4805 4806 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4807 4808 So the fiber has its own magic methods to read and write. if they would block, it registers 4809 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4810 4811 basically you issue the command and it tells you when it is done 4812 4813 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4814 4815 +/ 4816 4817 /++ 4818 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4819 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4820 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4821 4822 History: 4823 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4824 +/ 4825 version(cgi_use_fiber) 4826 __gshared size_t fiberStackSize = 4096 * 100; 4827 4828 version(cgi_use_fiber) 4829 class CgiFiber : Fiber { 4830 private void function(Socket) f_handler; 4831 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4832 f_handler(s); 4833 } 4834 this(void function(Socket) handler) { 4835 this.f_handler = handler; 4836 this(&f_handler_dg); 4837 } 4838 4839 this(void delegate(Socket) handler) { 4840 this.handler = handler; 4841 super(&run, fiberStackSize); 4842 } 4843 4844 Socket connection; 4845 void delegate(Socket) handler; 4846 4847 void run() { 4848 handler(connection); 4849 } 4850 4851 void delegate() postYield; 4852 4853 private void setPostYield(scope void delegate() py) @nogc { 4854 postYield = cast(void delegate()) py; 4855 } 4856 4857 void proceed() { 4858 try { 4859 call(); 4860 auto py = postYield; 4861 postYield = null; 4862 if(py !is null) 4863 py(); 4864 } catch(Exception e) { 4865 if(connection) 4866 connection.close(); 4867 goto terminate; 4868 } 4869 4870 if(state == State.TERM) { 4871 terminate: 4872 import core.memory; 4873 GC.removeRoot(cast(void*) this); 4874 } 4875 } 4876 } 4877 4878 version(cgi_use_fiber) 4879 version(Windows) { 4880 4881 extern(Windows) private { 4882 4883 import core.sys.windows.mswsock; 4884 4885 alias GROUP=uint; 4886 alias LPWSAPROTOCOL_INFOW = void*; 4887 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4888 alias WSASend = arsd.core.WSASend; 4889 alias WSARecv = arsd.core.WSARecv; 4890 alias WSABUF = arsd.core.WSABUF; 4891 4892 /+ 4893 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4894 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4895 4896 struct WSABUF { 4897 ULONG len; 4898 CHAR *buf; 4899 } 4900 +/ 4901 alias LPWSABUF = WSABUF*; 4902 4903 alias WSAOVERLAPPED = OVERLAPPED; 4904 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4905 /+ 4906 4907 alias LPFN_ACCEPTEX = 4908 BOOL 4909 function( 4910 SOCKET sListenSocket, 4911 SOCKET sAcceptSocket, 4912 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4913 void* lpOutputBuffer, 4914 WORD dwReceiveDataLength, 4915 WORD dwLocalAddressLength, 4916 WORD dwRemoteAddressLength, 4917 LPDWORD lpdwBytesReceived, 4918 LPOVERLAPPED lpOverlapped 4919 ); 4920 4921 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4922 +/ 4923 4924 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4925 } 4926 4927 private class PseudoblockingOverlappedSocket : Socket { 4928 SOCKET handle; 4929 4930 CgiFiber fiber; 4931 4932 this(AddressFamily af, SocketType st) { 4933 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4934 if(!handle) 4935 throw new Exception("WSASocketW"); 4936 this.handle = handle; 4937 4938 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4939 4940 if(iocp is null) { 4941 writeln(GetLastError()); 4942 throw new Exception("CreateIoCompletionPort"); 4943 } 4944 4945 super(cast(socket_t) handle, af); 4946 } 4947 this() pure nothrow @trusted { assert(0); } 4948 4949 override void blocking(bool) {} // meaningless to us, just ignore it. 4950 4951 protected override Socket accepting() pure nothrow { 4952 assert(0); 4953 } 4954 4955 bool addressesParsed; 4956 Address la; 4957 Address ra; 4958 4959 private void populateAddresses() { 4960 if(addressesParsed) 4961 return; 4962 addressesParsed = true; 4963 4964 int lalen, ralen; 4965 4966 sockaddr_in* la; 4967 sockaddr_in* ra; 4968 4969 lpfnGetAcceptExSockaddrs( 4970 scratchBuffer.ptr, 4971 0, // same as in the AcceptEx call! 4972 sockaddr_in.sizeof + 16, 4973 sockaddr_in.sizeof + 16, 4974 cast(sockaddr**) &la, 4975 &lalen, 4976 cast(sockaddr**) &ra, 4977 &ralen 4978 ); 4979 4980 if(la) 4981 this.la = new InternetAddress(*la); 4982 if(ra) 4983 this.ra = new InternetAddress(*ra); 4984 4985 } 4986 4987 override @property @trusted Address localAddress() { 4988 populateAddresses(); 4989 return la; 4990 } 4991 override @property @trusted Address remoteAddress() { 4992 populateAddresses(); 4993 return ra; 4994 } 4995 4996 PseudoblockingOverlappedSocket accepted; 4997 4998 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4999 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 5000 5001 override Socket accept() @trusted { 5002 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 5003 5004 if(lpfnAcceptEx is null) { 5005 DWORD dwBytes; 5006 GUID GuidAcceptEx = WSAID_ACCEPTEX; 5007 5008 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 5009 &GuidAcceptEx, GuidAcceptEx.sizeof, 5010 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 5011 &dwBytes, null, null); 5012 5013 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 5014 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 5015 &GuidAcceptEx, GuidAcceptEx.sizeof, 5016 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 5017 &dwBytes, null, null); 5018 5019 } 5020 5021 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 5022 accepted = pfa; 5023 5024 SOCKET pendingForAccept = pfa.handle; 5025 DWORD ignored; 5026 5027 auto ret = lpfnAcceptEx(handle, 5028 pendingForAccept, 5029 // buffer to receive up front 5030 pfa.scratchBuffer.ptr, 5031 0, 5032 // size of local and remote addresses. normally + 16. 5033 sockaddr_in.sizeof + 16, 5034 sockaddr_in.sizeof + 16, 5035 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 5036 &overlapped 5037 ); 5038 5039 return pfa; 5040 } 5041 5042 override void connect(Address to) { assert(0); } 5043 5044 DWORD lastAnswer; 5045 ubyte[1024] scratchBuffer; 5046 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 5047 5048 WSABUF[1] buffer; 5049 OVERLAPPED overlapped; 5050 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 5051 overlapped = overlapped.init; 5052 buffer[0].len = cast(DWORD) buf.length; 5053 buffer[0].buf = cast(ubyte*) buf.ptr; 5054 fiber.setPostYield( () { 5055 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 5056 if(GetLastError() != 997) { 5057 //throw new Exception("WSASend fail"); 5058 } 5059 } 5060 }); 5061 5062 Fiber.yield(); 5063 return lastAnswer; 5064 } 5065 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 5066 overlapped = overlapped.init; 5067 buffer[0].len = cast(DWORD) buf.length; 5068 buffer[0].buf = cast(ubyte*) buf.ptr; 5069 5070 DWORD flags2 = 0; 5071 5072 fiber.setPostYield(() { 5073 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 5074 if(GetLastError() != 997) { 5075 //writeln("WSARecv ", WSAGetLastError()); 5076 //throw new Exception("WSARecv fail"); 5077 } 5078 } 5079 }); 5080 5081 Fiber.yield(); 5082 return lastAnswer; 5083 } 5084 5085 // I might go back and implement these for udp things. 5086 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 5087 assert(0); 5088 } 5089 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 5090 assert(0); 5091 } 5092 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 5093 assert(0); 5094 } 5095 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 5096 assert(0); 5097 } 5098 5099 // lol overload sets 5100 alias send = typeof(super).send; 5101 alias receive = typeof(super).receive; 5102 alias sendTo = typeof(super).sendTo; 5103 alias receiveFrom = typeof(super).receiveFrom; 5104 5105 } 5106 } 5107 5108 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 5109 assert(connection !is null); 5110 version(cgi_use_fiber) { 5111 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 5112 5113 version(Windows) { 5114 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 5115 } 5116 5117 import core.memory; 5118 GC.addRoot(cast(void*) fiber); 5119 fiber.connection = connection; 5120 fiber.proceed(); 5121 } else { 5122 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 5123 } 5124 } 5125 5126 /+ 5127 5128 /+ 5129 The represents a recyclable per-task arena allocator. The default is to let the GC manage the whole block as a large array, meaning if a reference into it is escaped, it waste memory but is not dangerous. If you don't escape any references to it and don't do anything special, the GC collects it. 5130 5131 But, if you call `cgi.recyclable = true`, the memory is retained for the next request on the thread. If a reference is escaped, it is the user's problem; it can be modified (and break the `immutable` guarantees!) and thus be memory unsafe. They're taking responsibility for doing it right when they call `escape`. But if they do it right and opt into recycling, the memory is all reused to give a potential boost without requiring the GC's involvement. 5132 5133 What if one request used an abnormally large amount of memory though? Will recycling it keep that pinned forever? No, that's why it keeps track of some stats. If a working set was significantly above average and not fully utilized for a while, it will just let the GC have it again despite your suggestion to recycle it. 5134 5135 Be warned that growing the memory block may release the old, smaller block for garbage collection. If you retained references to it, it may not be collectable and lead to some unnecessary memory growth. It is probably best to try to keep the things sized in a continuous block that doesn't have to grow often. 5136 5137 Internally, it is broken up into a few blocks: 5138 * the request block. This holds the incoming request and associated data (parsed headers, variables, etc). 5139 * the scannable block. this holds pointers arrays, classes, etc. associated with this request, so named because the GC scans it. 5140 * the response block. This holds the output buffer. 5141 5142 And I may add more later if I decide to open this up to outside user code. 5143 5144 The scannable block is separate to limit the amount of work the GC has to do; no point asking it to scan that which need not be scanned. 5145 5146 The request and response blocks are separated because they will have different typical sizes, with the request likely being less predictable. Being able to release one to the GC while recycling the other might help, and having them grow independently (if needed) may also prevent some pain. 5147 5148 All of this are internal implementation details subject to change at any time without notice. It is valid for my recycle method to do absolutely nothing; the GC also eventually recycles memory! 5149 5150 Each active task can have its own recyclable memory object. When you recycle it, it is added to a thread-local freelist. If the list is excessively large, entries maybe discarded at random and left for the GC to prevent a temporary burst of activity from leading to a permanent waste of memory. 5151 +/ 5152 struct RecyclableMemory { 5153 private ubyte[] inputBuffer; 5154 private ubyte[] processedRequestBlock; 5155 private void[] scannableBlock; 5156 private ubyte[] outputBuffer; 5157 5158 RecyclableMemory* next; 5159 } 5160 5161 /++ 5162 This emulates the D associative array interface with a different internal implementation. 5163 5164 string s = cgi.get["foo"]; // just does cgi.getArray[x][$-1]; 5165 string[] arr = cgi.getArray["foo"]; 5166 5167 "foo" in cgi.get 5168 5169 foreach(k, v; cgi.get) 5170 5171 cgi.get.toAA // for compatibility 5172 5173 // and this can urldecode lazily tbh... in-place even, since %xx is always longer than a single char thing it turns into... 5174 ... but how does it mark that it has already been processed in-place? it'd have to just add it to the index then. 5175 5176 deprecated alias toAA this; 5177 +/ 5178 struct VariableCollection { 5179 private VariableArrayCollection* vac; 5180 5181 const(char[]) opIndex(scope const char[] key) { 5182 return (*vac)[key][$-1]; 5183 } 5184 5185 const(char[]*) opBinaryRight(string op : "in")(scope const char[] key) { 5186 return key in (*vac); 5187 } 5188 5189 int opApply(int delegate(scope const(char)[] key, scope const(char)[] value) dg) { 5190 foreach(k, v; *vac) { 5191 if(auto res = dg(k, v[$-1])) 5192 return res; 5193 } 5194 return 0; 5195 } 5196 5197 immutable(string[string]) toAA() { 5198 string[string] aa; 5199 foreach(k, v; *vac) 5200 aa[k.idup] = v[$-1].idup; 5201 return aa; 5202 } 5203 5204 deprecated alias toAA this; 5205 } 5206 5207 struct VariableArrayCollection { 5208 /+ 5209 This needs the actual implementation of looking it up. As it pulls data, it should 5210 decode and index for later. 5211 5212 The index will go into a block attached to the cgi object and it should prolly be sorted 5213 something like 5214 5215 [count of names] 5216 [slice to name][count of values][slice to value, decoded in-place, ...] 5217 ... 5218 +/ 5219 private Cgi cgi; 5220 5221 const(char[][]) opIndex(scope const char[] key) { 5222 return null; 5223 } 5224 5225 const(char[][]*) opBinaryRight(string op : "in")(scope const char[] key) { 5226 return null; 5227 } 5228 5229 // int opApply(int delegate(scope const(char)[] key, scope const(char)[][] value) dg) 5230 5231 immutable(string[string]) toAA() { 5232 return null; 5233 } 5234 5235 deprecated alias toAA this; 5236 5237 } 5238 5239 struct HeaderCollection { 5240 5241 } 5242 +/ 5243 5244 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 5245 scope(failure) { 5246 // catch all for other errors 5247 try { 5248 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 5249 connection.close(); 5250 } catch(Exception e) {} // swallow it, we're aborting anyway. 5251 } 5252 5253 bool closeConnection = alwaysCloseConnection; 5254 5255 /+ 5256 ubyte[4096] inputBuffer = void; 5257 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5258 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5259 5260 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5261 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5262 ir.__ctor(connection, inputBuffer[], true); 5263 +/ 5264 5265 auto ir = new BufferedInputRange(connection); 5266 5267 while(!ir.empty) { 5268 5269 if(ir.view.length == 0) { 5270 ir.popFront(); 5271 if(ir.sourceClosed) { 5272 connection.close(); 5273 closeConnection = true; 5274 break; 5275 } 5276 } 5277 5278 Cgi cgi; 5279 try { 5280 cgi = new CustomCgi(ir, &closeConnection); 5281 // There's a bunch of these casts around because the type matches up with 5282 // the -version=.... specifiers, just you can also create a RequestServer 5283 // and instantiate the things where the types don't match up. It isn't exactly 5284 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5285 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5286 } catch(ConnectionClosedException ce) { 5287 closeConnection = true; 5288 break; 5289 } catch(ConnectionException ce) { 5290 // broken pipe or something, just abort the connection 5291 closeConnection = true; 5292 break; 5293 } catch(HttpVersionNotSupportedException ve) { 5294 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5295 closeConnection = true; 5296 break; 5297 } catch(Throwable t) { 5298 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5299 // anyway let's kill the connection 5300 version(CRuntime_Musl) { 5301 stderr.rawWrite(t.toString()); 5302 stderr.rawWrite("\n"); 5303 } else { 5304 stderr.writeln(t.toString()); 5305 } 5306 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5307 closeConnection = true; 5308 break; 5309 } 5310 assert(cgi !is null); 5311 scope(exit) 5312 cgi.dispose(); 5313 5314 try { 5315 fun(cgi); 5316 cgi.close(); 5317 if(cgi.websocketMode) 5318 closeConnection = true; 5319 } catch(AuthorizationRequiredException are) { 5320 cgi.setResponseStatus("401 Authorization Required"); 5321 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 5322 cgi.close(); 5323 } catch(ConnectionException ce) { 5324 // broken pipe or something, just abort the connection 5325 closeConnection = true; 5326 } catch(ConnectionClosedException ce) { 5327 // broken pipe or something, just abort the connection 5328 closeConnection = true; 5329 } catch(Throwable t) { 5330 // a processing error can be recovered from 5331 version(CRuntime_Musl) {} else 5332 stderr.writeln(t.toString); 5333 if(!handleException(cgi, t)) 5334 closeConnection = true; 5335 } 5336 5337 if(globalStopFlag) 5338 closeConnection = true; 5339 5340 if(closeConnection || alwaysCloseConnection) { 5341 connection.shutdown(SocketShutdown.BOTH); 5342 connection.close(); 5343 ir.dispose(); 5344 closeConnection = false; // don't reclose after loop 5345 break; 5346 } else { 5347 if(ir.front.length) { 5348 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5349 } else if(ir.sourceClosed) { 5350 ir.source.shutdown(SocketShutdown.BOTH); 5351 ir.source.close(); 5352 ir.dispose(); 5353 closeConnection = false; 5354 } else { 5355 continue; 5356 // break; // this was for a keepalive experiment 5357 } 5358 } 5359 } 5360 5361 if(closeConnection) { 5362 connection.shutdown(SocketShutdown.BOTH); 5363 connection.close(); 5364 ir.dispose(); 5365 } 5366 5367 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5368 } 5369 5370 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5371 // and now we can buffer 5372 scope(failure) 5373 connection.close(); 5374 5375 import al = std.algorithm; 5376 5377 size_t size; 5378 5379 string[string] headers; 5380 5381 auto range = new BufferedInputRange(connection); 5382 more_data: 5383 auto chunk = range.front(); 5384 // waiting for colon for header length 5385 auto idx = indexOf(cast(string) chunk, ':'); 5386 if(idx == -1) { 5387 try { 5388 range.popFront(); 5389 } catch(Exception e) { 5390 // it is just closed, no big deal 5391 connection.close(); 5392 return; 5393 } 5394 goto more_data; 5395 } 5396 5397 size = to!size_t(cast(string) chunk[0 .. idx]); 5398 chunk = range.consume(idx + 1); 5399 // reading headers 5400 if(chunk.length < size) 5401 range.popFront(0, size + 1); 5402 // we are now guaranteed to have enough 5403 chunk = range.front(); 5404 assert(chunk.length > size); 5405 5406 idx = 0; 5407 string key; 5408 string value; 5409 foreach(part; al.splitter(chunk, '\0')) { 5410 if(idx & 1) { // odd is value 5411 value = cast(string)(part.idup); 5412 headers[key] = value; // commit 5413 } else 5414 key = cast(string)(part.idup); 5415 idx++; 5416 } 5417 5418 enforce(chunk[size] == ','); // the terminator 5419 5420 range.consume(size + 1); 5421 // reading data 5422 // this will be done by Cgi 5423 5424 const(ubyte)[] getScgiChunk() { 5425 // we are already primed 5426 auto data = range.front(); 5427 if(data.length == 0 && !range.sourceClosed) { 5428 range.popFront(0); 5429 data = range.front(); 5430 } else if (range.sourceClosed) 5431 range.source.close(); 5432 5433 return data; 5434 } 5435 5436 void writeScgi(const(ubyte)[] data) { 5437 sendAll(connection, data); 5438 } 5439 5440 void flushScgi() { 5441 // I don't *think* I have to do anything.... 5442 } 5443 5444 Cgi cgi; 5445 try { 5446 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5447 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5448 } catch(Throwable t) { 5449 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5450 connection.close(); 5451 return; // this connection is dead 5452 } 5453 assert(cgi !is null); 5454 scope(exit) cgi.dispose(); 5455 try { 5456 fun(cgi); 5457 cgi.close(); 5458 connection.close(); 5459 5460 } catch(AuthorizationRequiredException are) { 5461 cgi.setResponseStatus("401 Authorization Required"); 5462 cgi.header ("WWW-Authenticate: "~are.type~" realm=\""~are.realm~"\""); 5463 cgi.close(); 5464 } catch(Throwable t) { 5465 // no std err 5466 if(!handleException(cgi, t)) { 5467 connection.close(); 5468 return; 5469 } else { 5470 connection.close(); 5471 return; 5472 } 5473 } 5474 } 5475 5476 string printDate(DateTime date) { 5477 char[29] buffer = void; 5478 printDateToBuffer(date, buffer[]); 5479 return buffer.idup; 5480 } 5481 5482 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5483 assert(buffer.length >= 29); 5484 // 29 static length ? 5485 5486 static immutable daysOfWeek = [ 5487 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5488 ]; 5489 5490 static immutable months = [ 5491 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5492 ]; 5493 5494 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5495 buffer[3 .. 5] = ", "; 5496 buffer[5] = date.day / 10 + '0'; 5497 buffer[6] = date.day % 10 + '0'; 5498 buffer[7] = ' '; 5499 buffer[8 .. 11] = months[date.month]; 5500 buffer[11] = ' '; 5501 auto y = date.year; 5502 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5503 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5504 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5505 buffer[15] = cast(char) (y + '0'); 5506 buffer[16] = ' '; 5507 buffer[17] = date.hour / 10 + '0'; 5508 buffer[18] = date.hour % 10 + '0'; 5509 buffer[19] = ':'; 5510 buffer[20] = date.minute / 10 + '0'; 5511 buffer[21] = date.minute % 10 + '0'; 5512 buffer[22] = ':'; 5513 buffer[23] = date.second / 10 + '0'; 5514 buffer[24] = date.second % 10 + '0'; 5515 buffer[25 .. $] = " GMT"; 5516 5517 return 29; 5518 } 5519 5520 5521 // Referencing this gigantic typeid seems to remind the compiler 5522 // to actually put the symbol in the object file. I guess the immutable 5523 // assoc array array isn't actually included in druntime 5524 void hackAroundLinkerError() { 5525 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5526 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5527 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5528 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5529 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5530 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5531 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5532 // this is getting kinda ridiculous btw. Moving assoc arrays 5533 // to the library is the pain that keeps on coming. 5534 5535 // eh this broke the build on the work server 5536 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5537 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5538 } 5539 5540 5541 5542 5543 5544 version(fastcgi) { 5545 pragma(lib, "fcgi"); 5546 5547 static if(size_t.sizeof == 8) // 64 bit 5548 alias long c_int; 5549 else 5550 alias int c_int; 5551 5552 extern(C) { 5553 struct FCGX_Stream { 5554 ubyte* rdNext; 5555 ubyte* wrNext; 5556 ubyte* stop; 5557 ubyte* stopUnget; 5558 c_int isReader; 5559 c_int isClosed; 5560 c_int wasFCloseCalled; 5561 c_int FCGI_errno; 5562 void* function(FCGX_Stream* stream) fillBuffProc; 5563 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5564 void* data; 5565 } 5566 5567 // note: this is meant to be opaque, so don't access it directly 5568 struct FCGX_Request { 5569 int requestId; 5570 int role; 5571 FCGX_Stream* inStream; 5572 FCGX_Stream* outStream; 5573 FCGX_Stream* errStream; 5574 char** envp; 5575 void* paramsPtr; 5576 int ipcFd; 5577 int isBeginProcessed; 5578 int keepConnection; 5579 int appStatus; 5580 int nWriters; 5581 int flags; 5582 int listen_sock; 5583 } 5584 5585 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5586 void FCGX_Init(); 5587 5588 int FCGX_Accept_r(FCGX_Request *request); 5589 5590 5591 alias char** FCGX_ParamArray; 5592 5593 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5594 c_int FCGX_GetChar(FCGX_Stream* stream); 5595 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5596 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5597 c_int FCGX_FFlush(FCGX_Stream *stream); 5598 5599 int FCGX_OpenSocket(in char*, int); 5600 } 5601 } 5602 5603 5604 /* This might go int a separate module eventually. It is a network input helper class. */ 5605 5606 import std.socket; 5607 5608 version(cgi_use_fiber) { 5609 import core.thread; 5610 5611 version(linux) { 5612 import core.sys.linux.epoll; 5613 5614 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5615 } else version(Windows) { 5616 // declaring the iocp thing below... 5617 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5618 } 5619 5620 version(Windows) 5621 __gshared HANDLE iocp; 5622 5623 version(cgi_use_fiber) { 5624 version(linux) 5625 private enum WakeupEvent { 5626 Read = EPOLLIN, 5627 Write = EPOLLOUT 5628 } 5629 else version(Windows) 5630 private enum WakeupEvent { 5631 Read, Write 5632 } 5633 else static assert(0); 5634 } 5635 5636 version(cgi_use_fiber) 5637 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5638 5639 // static cast since I know what i have in here and don't want to pay for dynamic cast 5640 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5641 5642 version(linux) { 5643 f.setPostYield = () { 5644 if(*registered) { 5645 // rearm 5646 epoll_event evt; 5647 evt.events = e | EPOLLONESHOT; 5648 evt.data.ptr = cast(void*) f; 5649 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5650 throw new Exception("epoll_ctl"); 5651 } else { 5652 // initial registration 5653 *registered = true ; 5654 int fd = source.handle; 5655 epoll_event evt; 5656 evt.events = e | EPOLLONESHOT; 5657 evt.data.ptr = cast(void*) f; 5658 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5659 throw new Exception("epoll_ctl"); 5660 } 5661 }; 5662 5663 Fiber.yield(); 5664 5665 f.setPostYield(null); 5666 } else version(Windows) { 5667 Fiber.yield(); 5668 } 5669 else static assert(0); 5670 } 5671 5672 version(cgi_use_fiber) 5673 void unregisterSource(Socket s) { 5674 version(linux) { 5675 epoll_event evt; 5676 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5677 } else version(Windows) { 5678 // intentionally blank 5679 } 5680 else static assert(0); 5681 } 5682 5683 // it is a class primarily for reference semantics 5684 // I might change this interface 5685 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5686 class BufferedInputRange { 5687 version(Posix) 5688 this(int source, ubyte[] buffer = null) { 5689 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5690 } 5691 5692 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5693 // if they connect but never send stuff to us, we don't want it wasting the process 5694 // so setting a time out 5695 version(cgi_use_fiber) 5696 source.blocking = false; 5697 else 5698 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5699 5700 this.source = source; 5701 if(buffer is null) { 5702 underlyingBuffer = new ubyte[4096]; 5703 this.allowGrowth = true; 5704 } else { 5705 underlyingBuffer = buffer; 5706 this.allowGrowth = allowGrowth; 5707 } 5708 5709 assert(underlyingBuffer.length); 5710 5711 // we assume view.ptr is always inside underlyingBuffer 5712 view = underlyingBuffer[0 .. 0]; 5713 5714 popFront(); // prime 5715 } 5716 5717 version(cgi_use_fiber) { 5718 bool registered; 5719 } 5720 5721 void dispose() { 5722 version(cgi_use_fiber) { 5723 if(registered) 5724 unregisterSource(source); 5725 } 5726 } 5727 5728 /** 5729 A slight difference from regular ranges is you can give it the maximum 5730 number of bytes to consume. 5731 5732 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5733 consume() yourself and use a regular foreach, it will infinitely loop! 5734 5735 The default is to do what a normal range does, and consume the whole buffer 5736 and wait for additional input. 5737 5738 You can also specify 0, to append to the buffer, or any other number 5739 to remove the front n bytes and wait for more. 5740 */ 5741 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5742 if(sourceClosed) 5743 throw new ConnectionClosedException("can't get any more data from a closed source"); 5744 if(!skipConsume) 5745 consume(maxBytesToConsume); 5746 5747 // we might have to grow the buffer 5748 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5749 if(allowGrowth) { 5750 //import std.stdio; writeln("growth"); 5751 auto viewStart = view.ptr - underlyingBuffer.ptr; 5752 size_t growth = 4096; 5753 // make sure we have enough for what we're being asked for 5754 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5755 growth = minBytesToSettleFor - underlyingBuffer.length; 5756 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5757 underlyingBuffer.length += growth; 5758 view = underlyingBuffer[viewStart .. view.length]; 5759 } else 5760 throw new Exception("No room left in the buffer"); 5761 } 5762 5763 do { 5764 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5765 try_again: 5766 auto ret = source.receive(freeSpace); 5767 if(ret == Socket.ERROR) { 5768 if(wouldHaveBlocked()) { 5769 version(cgi_use_fiber) { 5770 registerEventWakeup(®istered, source, WakeupEvent.Read); 5771 goto try_again; 5772 } else { 5773 // gonna treat a timeout here as a close 5774 sourceClosed = true; 5775 return; 5776 } 5777 } 5778 version(Posix) { 5779 import core.stdc.errno; 5780 if(errno == EINTR || errno == EAGAIN) { 5781 goto try_again; 5782 } 5783 if(errno == ECONNRESET) { 5784 sourceClosed = true; 5785 return; 5786 } 5787 } 5788 throw new Exception(lastSocketError); // FIXME 5789 } 5790 if(ret == 0) { 5791 sourceClosed = true; 5792 return; 5793 } 5794 5795 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5796 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5797 //import std.stdio; writeln(cast(string) view); 5798 } while(view.length < minBytesToSettleFor); 5799 } 5800 5801 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5802 /// You might want to idup the data you are consuming if you store it, since it may 5803 /// be overwritten on the new popFront. 5804 /// 5805 /// You do not need to call this if you always want to wait for more data when you 5806 /// consume some. 5807 ubyte[] consume(size_t bytes) { 5808 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5809 view = view[bytes > $ ? $ : bytes .. $]; 5810 if(view.length == 0) { 5811 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5812 /* 5813 writeln("HERE"); 5814 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5815 writeln(cast(string)front); 5816 writeln("DONE"); 5817 */ 5818 } 5819 return front; 5820 } 5821 5822 bool empty() { 5823 return sourceClosed && view.length == 0; 5824 } 5825 5826 ubyte[] front() { 5827 return view; 5828 } 5829 5830 invariant() { 5831 assert(view.ptr >= underlyingBuffer.ptr); 5832 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5833 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5834 } 5835 5836 ubyte[] underlyingBuffer; 5837 bool allowGrowth; 5838 ubyte[] view; 5839 Socket source; 5840 bool sourceClosed; 5841 } 5842 5843 private class FakeSocketForStdin : Socket { 5844 import std.stdio; 5845 5846 this() { 5847 5848 } 5849 5850 private bool closed; 5851 5852 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5853 if(closed) 5854 throw new Exception("Closed"); 5855 return stdin.rawRead(buffer).length; 5856 } 5857 5858 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5859 if(closed) 5860 throw new Exception("Closed"); 5861 stdout.rawWrite(buffer); 5862 return buffer.length; 5863 } 5864 5865 override void close() @trusted scope { 5866 (cast(void delegate() @nogc nothrow) &realClose)(); 5867 } 5868 5869 override void shutdown(SocketShutdown s) { 5870 // FIXME 5871 } 5872 5873 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5874 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5875 5876 override @property @trusted Address remoteAddress() { return null; } 5877 override @property @trusted Address localAddress() { return null; } 5878 5879 void realClose() { 5880 closed = true; 5881 try { 5882 stdin.close(); 5883 stdout.close(); 5884 } catch(Exception e) { 5885 5886 } 5887 } 5888 } 5889 5890 import core.sync.semaphore; 5891 import core.atomic; 5892 5893 /** 5894 To use this thing: 5895 5896 --- 5897 void handler(Socket s) { do something... } 5898 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5899 manager.listen(); 5900 --- 5901 5902 The 4th parameter is optional. 5903 5904 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5905 comes in, you will get control. You can just continue; though to fetch more. 5906 5907 5908 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5909 */ 5910 class ListeningConnectionManager { 5911 Semaphore semaphore; 5912 Socket[256] queue; 5913 shared(ubyte) nextIndexFront; 5914 ubyte nextIndexBack; 5915 shared(int) queueLength; 5916 5917 Socket acceptCancelable() { 5918 version(Posix) { 5919 import core.sys.posix.sys.select; 5920 fd_set read_fds; 5921 FD_ZERO(&read_fds); 5922 int max = 0; 5923 foreach(listener; listeners) { 5924 FD_SET(listener.handle, &read_fds); 5925 if(listener.handle > max) 5926 max = listener.handle; 5927 } 5928 if(cancelfd != -1) { 5929 FD_SET(cancelfd, &read_fds); 5930 if(cancelfd > max) 5931 max = cancelfd; 5932 } 5933 auto ret = select(max + 1, &read_fds, null, null, null); 5934 if(ret == -1) { 5935 import core.stdc.errno; 5936 if(errno == EINTR) 5937 return null; 5938 else 5939 throw new Exception("wtf select"); 5940 } 5941 5942 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5943 return null; 5944 } 5945 5946 foreach(listener; listeners) { 5947 if(FD_ISSET(listener.handle, &read_fds)) 5948 return listener.accept(); 5949 } 5950 5951 return null; 5952 } else { 5953 5954 auto check = new SocketSet(); 5955 5956 keep_looping: 5957 check.reset(); 5958 foreach(listener; listeners) 5959 check.add(listener); 5960 5961 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5962 auto got = Socket.select(check, null, null, 3.seconds); 5963 if(got > 0) 5964 foreach(listener; listeners) 5965 if(check.isSet(listener)) 5966 return listener.accept(); 5967 if(globalStopFlag) 5968 return null; 5969 else 5970 goto keep_looping; 5971 } 5972 } 5973 5974 int defaultNumberOfThreads() { 5975 import std.parallelism; 5976 version(cgi_use_fiber) { 5977 return totalCPUs * 2 + 1; // still chance some will be pointlessly blocked anyway 5978 } else { 5979 // I times 4 here because there's a good chance some will be blocked on i/o. 5980 return totalCPUs * 4; 5981 } 5982 5983 } 5984 5985 void listen() { 5986 shared(int) loopBroken; 5987 5988 version(Posix) { 5989 import core.sys.posix.signal; 5990 signal(SIGPIPE, SIG_IGN); 5991 } 5992 5993 version(linux) { 5994 if(cancelfd == -1) 5995 cancelfd = eventfd(0, 0); 5996 } 5997 5998 version(cgi_no_threads) { 5999 // NEVER USE THIS 6000 // it exists only for debugging and other special occasions 6001 6002 // the thread mode is faster and less likely to stall the whole 6003 // thing when a request is slow 6004 while(!loopBroken && !globalStopFlag) { 6005 auto sn = acceptCancelable(); 6006 if(sn is null) continue; 6007 cloexec(sn); 6008 try { 6009 handler(sn); 6010 } catch(Exception e) { 6011 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 6012 sn.close(); 6013 } 6014 } 6015 } else { 6016 6017 if(useFork) { 6018 version(linux) { 6019 //asm { int 3; } 6020 fork(); 6021 } 6022 } 6023 6024 version(cgi_use_fiber) { 6025 6026 version(Windows) { 6027 // please note these are overlapped sockets! so the accept just kicks things off 6028 foreach(listener; listeners) 6029 listener.accept(); 6030 } 6031 6032 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 6033 foreach(i, ref thread; threads) { 6034 thread = new WorkerThread(this, handler, cast(int) i); 6035 thread.start(); 6036 } 6037 6038 bool fiber_crash_check() { 6039 bool hasAnyRunning; 6040 foreach(thread; threads) { 6041 if(!thread.isRunning) { 6042 thread.join(); 6043 } else hasAnyRunning = true; 6044 } 6045 6046 return (!hasAnyRunning); 6047 } 6048 6049 6050 while(!globalStopFlag) { 6051 Thread.sleep(1.seconds); 6052 if(fiber_crash_check()) 6053 break; 6054 } 6055 6056 } else { 6057 semaphore = new Semaphore(); 6058 6059 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 6060 foreach(i, ref thread; threads) { 6061 thread = new ConnectionThread(this, handler, cast(int) i); 6062 thread.start(); 6063 } 6064 6065 while(!loopBroken && !globalStopFlag) { 6066 Socket sn; 6067 6068 bool crash_check() { 6069 bool hasAnyRunning; 6070 foreach(thread; threads) { 6071 if(!thread.isRunning) { 6072 thread.join(); 6073 } else hasAnyRunning = true; 6074 } 6075 6076 return (!hasAnyRunning); 6077 } 6078 6079 6080 void accept_new_connection() { 6081 sn = acceptCancelable(); 6082 if(sn is null) return; 6083 cloexec(sn); 6084 if(tcp) { 6085 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6086 // on the socket because we do some buffering internally. I think this helps, 6087 // certainly does for small requests, and I think it does for larger ones too 6088 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6089 6090 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6091 } 6092 } 6093 6094 void existing_connection_new_data() { 6095 // wait until a slot opens up 6096 // int waited = 0; 6097 while(queueLength >= queue.length) { 6098 Thread.sleep(1.msecs); 6099 // waited ++; 6100 } 6101 // if(waited) {import std.stdio; writeln(waited);} 6102 synchronized(this) { 6103 queue[nextIndexBack] = sn; 6104 nextIndexBack++; 6105 atomicOp!"+="(queueLength, 1); 6106 } 6107 semaphore.notify(); 6108 } 6109 6110 6111 accept_new_connection(); 6112 if(sn !is null) 6113 existing_connection_new_data(); 6114 else if(sn is null && globalStopFlag) { 6115 foreach(thread; threads) { 6116 semaphore.notify(); 6117 } 6118 Thread.sleep(50.msecs); 6119 } 6120 6121 if(crash_check()) 6122 break; 6123 } 6124 } 6125 6126 // FIXME: i typically stop this with ctrl+c which never 6127 // actually gets here. i need to do a sigint handler. 6128 if(cleanup) 6129 cleanup(); 6130 } 6131 } 6132 6133 //version(linux) 6134 //int epoll_fd; 6135 6136 bool tcp; 6137 void delegate() cleanup; 6138 6139 private void function(Socket) fhandler; 6140 private void dg_handler(Socket s) { 6141 fhandler(s); 6142 } 6143 6144 6145 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6146 fhandler = handler; 6147 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 6148 } 6149 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6150 string[] host; 6151 ushort[] port; 6152 6153 foreach(spec; listenSpec) { 6154 /+ 6155 The format: 6156 6157 protocol:// 6158 address_spec 6159 6160 Protocol is optional. Must be http, https, scgi, or fastcgi. 6161 6162 address_spec is either: 6163 ipv4 address : port 6164 [ipv6 address] : port 6165 unix:filename 6166 abstract:name 6167 port <which is tcp but on any interface> 6168 +/ 6169 6170 string protocol; 6171 string address_spec; 6172 6173 auto protocolIdx = spec.indexOf("://"); 6174 if(protocolIdx != -1) { 6175 protocol = spec[0 .. protocolIdx]; 6176 address_spec = spec[protocolIdx + "://".length .. $]; 6177 } else { 6178 address_spec = spec; 6179 } 6180 6181 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 6182 host ~= address_spec; 6183 port ~= 0; 6184 } else { 6185 auto idx = address_spec.lastIndexOf(":"); 6186 if(idx == -1) { 6187 host ~= null; 6188 } else { 6189 auto as = address_spec[0 .. idx]; 6190 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 6191 as = as[1 .. $-1]; 6192 host ~= as; 6193 } 6194 port ~= address_spec[idx + 1 .. $].to!ushort; 6195 } 6196 6197 } 6198 6199 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 6200 } 6201 6202 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6203 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 6204 } 6205 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6206 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 6207 } 6208 6209 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6210 fhandler = handler; 6211 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 6212 } 6213 6214 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 6215 assert(host.length == port.length); 6216 6217 this.handler = handler; 6218 this.useFork = useFork; 6219 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 6220 6221 listeners.reserve(host.length); 6222 6223 foreach(i; 0 .. host.length) 6224 if(host[i] == "localhost") { 6225 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 6226 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 6227 } else { 6228 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 6229 } 6230 6231 version(cgi_use_fiber) 6232 if(useFork) { 6233 foreach(listener; listeners) 6234 listener.blocking = false; 6235 } 6236 6237 // this is the UI control thread and thus gets more priority 6238 Thread.getThis.priority = Thread.PRIORITY_MAX; 6239 } 6240 6241 Socket[] listeners; 6242 void delegate(Socket) handler; 6243 6244 immutable bool useFork; 6245 int numberOfThreads; 6246 } 6247 6248 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 6249 Socket listener; 6250 if(host.startsWith("unix:")) { 6251 version(Posix) { 6252 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6253 cloexec(listener); 6254 string filename = host["unix:".length .. $].idup; 6255 listener.bind(new UnixAddress(filename)); 6256 cleanup = delegate() { 6257 listener.close(); 6258 import std.file; 6259 remove(filename); 6260 }; 6261 tcp = false; 6262 } else { 6263 throw new Exception("unix sockets not supported on this system"); 6264 } 6265 } else if(host.startsWith("abstract:")) { 6266 version(linux) { 6267 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6268 cloexec(listener); 6269 string filename = "\0" ~ host["abstract:".length .. $]; 6270 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6271 listener.bind(new UnixAddress(filename)); 6272 tcp = false; 6273 } else { 6274 throw new Exception("abstract unix sockets not supported on this system"); 6275 } 6276 } else { 6277 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6278 version(cgi_use_fiber) { 6279 version(Windows) 6280 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6281 else 6282 listener = new Socket(address.addressFamily, SocketType.STREAM); 6283 } else { 6284 listener = new Socket(address.addressFamily, SocketType.STREAM); 6285 } 6286 cloexec(listener); 6287 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6288 if(address.addressFamily == AddressFamily.INET6) 6289 listener.setOption(SocketOptionLevel.IPV6, SocketOption.IPV6_V6ONLY, true); 6290 listener.bind(address); 6291 cleanup = delegate() { 6292 listener.close(); 6293 }; 6294 tcp = true; 6295 } 6296 6297 listener.listen(backQueue); 6298 6299 if (dropPrivs !is null) // can be null, backwards compatibility 6300 dropPrivs(); 6301 6302 return listener; 6303 } 6304 6305 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6306 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6307 if(data.length == 0) return; 6308 ptrdiff_t amount; 6309 //import std.stdio; writeln("***",cast(string) data,"///"); 6310 do { 6311 amount = s.send(data); 6312 if(amount == Socket.ERROR) { 6313 version(cgi_use_fiber) { 6314 if(wouldHaveBlocked()) { 6315 bool registered = true; 6316 registerEventWakeup(®istered, s, WakeupEvent.Write); 6317 continue; 6318 } 6319 } 6320 throw new ConnectionException(s, lastSocketError, file, line); 6321 } 6322 assert(amount > 0); 6323 6324 data = data[amount .. $]; 6325 } while(data.length); 6326 } 6327 6328 class ConnectionException : Exception { 6329 Socket socket; 6330 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6331 this.socket = s; 6332 super(msg, file, line); 6333 } 6334 } 6335 6336 class HttpVersionNotSupportedException : Exception { 6337 this(string file = __FILE__, size_t line = __LINE__) { 6338 super("HTTP Version Not Supported", file, line); 6339 } 6340 } 6341 6342 alias void delegate(Socket) CMT; 6343 6344 import core.thread; 6345 /+ 6346 cgi.d now uses a hybrid of event i/o and threads at the top level. 6347 6348 Top level thread is responsible for accepting sockets and selecting on them. 6349 6350 It then indicates to a child that a request is pending, and any random worker 6351 thread that is free handles it. It goes into blocking mode and handles that 6352 http request to completion. 6353 6354 At that point, it goes back into the waiting queue. 6355 6356 6357 This concept is only implemented on Linux. On all other systems, it still 6358 uses the worker threads and semaphores (which is perfectly fine for a lot of 6359 things! Just having a great number of keep-alive connections will break that.) 6360 6361 6362 So the algorithm is: 6363 6364 select(accept, event, pending) 6365 if accept -> send socket to free thread, if any. if not, add socket to queue 6366 if event -> send the signaling thread a socket from the queue, if not, mark it free 6367 - event might block until it can be *written* to. it is a fifo sending socket fds! 6368 6369 A worker only does one http request at a time, then signals its availability back to the boss. 6370 6371 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6372 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6373 actual FD will not be kept out here. 6374 6375 So: 6376 queue = sockets we know are ready to read now, but no worker thread is available 6377 idle list = worker threads not doing anything else. they signal back and forth 6378 6379 the workers all read off the event fd. This is the semaphore wait 6380 6381 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6382 it puts it in the queue and writes to the event fd. 6383 6384 The child could put the socket back in the epoll thing itself. 6385 6386 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6387 +/ 6388 class ConnectionThread : Thread { 6389 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6390 this.lcm = lcm; 6391 this.dg = dg; 6392 this.myThreadNumber = myThreadNumber; 6393 super(&run); 6394 } 6395 6396 void run() { 6397 while(true) { 6398 // so if there's a bunch of idle keep-alive connections, it can 6399 // consume all the worker threads... just sitting there. 6400 lcm.semaphore.wait(); 6401 if(globalStopFlag) 6402 return; 6403 Socket socket; 6404 synchronized(lcm) { 6405 auto idx = lcm.nextIndexFront; 6406 socket = lcm.queue[idx]; 6407 lcm.queue[idx] = null; 6408 atomicOp!"+="(lcm.nextIndexFront, 1); 6409 atomicOp!"-="(lcm.queueLength, 1); 6410 } 6411 try { 6412 //import std.stdio; writeln(myThreadNumber, " taking it"); 6413 dg(socket); 6414 /+ 6415 if(socket.isAlive) { 6416 // process it more later 6417 version(linux) { 6418 import core.sys.linux.epoll; 6419 epoll_event ev; 6420 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6421 ev.data.fd = socket.handle; 6422 import std.stdio; writeln("adding"); 6423 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6424 if(errno == EEXIST) { 6425 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6426 ev.data.fd = socket.handle; 6427 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6428 throw new Exception("epoll_ctl " ~ to!string(errno)); 6429 } else 6430 throw new Exception("epoll_ctl " ~ to!string(errno)); 6431 } 6432 //import std.stdio; writeln("keep alive"); 6433 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6434 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6435 } else { 6436 continue; // hope it times out in a reasonable amount of time... 6437 } 6438 } 6439 +/ 6440 } catch(ConnectionClosedException e) { 6441 // can just ignore this, it is fairly normal 6442 socket.close(); 6443 } catch(Throwable e) { 6444 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6445 socket.close(); 6446 } 6447 } 6448 } 6449 6450 ListeningConnectionManager lcm; 6451 CMT dg; 6452 int myThreadNumber; 6453 } 6454 6455 version(cgi_use_fiber) 6456 class WorkerThread : Thread { 6457 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6458 this.lcm = lcm; 6459 this.dg = dg; 6460 this.myThreadNumber = myThreadNumber; 6461 super(&run); 6462 } 6463 6464 version(Windows) 6465 void run() { 6466 auto timeout = INFINITE; 6467 PseudoblockingOverlappedSocket key; 6468 OVERLAPPED* overlapped; 6469 DWORD bytes; 6470 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6471 if(key is null) 6472 continue; 6473 key.lastAnswer = bytes; 6474 if(key.fiber) { 6475 key.fiber.proceed(); 6476 } else { 6477 // we have a new connection, issue the first receive on it and issue the next accept 6478 6479 auto sn = key.accepted; 6480 6481 key.accept(); 6482 6483 cloexec(sn); 6484 if(lcm.tcp) { 6485 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6486 // on the socket because we do some buffering internally. I think this helps, 6487 // certainly does for small requests, and I think it does for larger ones too 6488 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6489 6490 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6491 } 6492 6493 dg(sn); 6494 } 6495 } 6496 //SleepEx(INFINITE, TRUE); 6497 } 6498 6499 version(linux) 6500 void run() { 6501 6502 import core.sys.linux.epoll; 6503 epfd = epoll_create1(EPOLL_CLOEXEC); 6504 if(epfd == -1) 6505 throw new Exception("epoll_create1 " ~ to!string(errno)); 6506 scope(exit) { 6507 import core.sys.posix.unistd; 6508 close(epfd); 6509 } 6510 6511 { 6512 epoll_event ev; 6513 ev.events = EPOLLIN; 6514 ev.data.fd = cancelfd; 6515 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6516 } 6517 6518 foreach(listener; lcm.listeners) { 6519 epoll_event ev; 6520 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6521 ev.data.fd = listener.handle; 6522 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6523 throw new Exception("epoll_ctl " ~ to!string(errno)); 6524 } 6525 6526 6527 6528 while(!globalStopFlag) { 6529 Socket sn; 6530 6531 epoll_event[64] events; 6532 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6533 if(nfds == -1) { 6534 if(errno == EINTR) 6535 continue; 6536 throw new Exception("epoll_wait " ~ to!string(errno)); 6537 } 6538 6539 outer: foreach(idx; 0 .. nfds) { 6540 auto flags = events[idx].events; 6541 6542 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6543 globalStopFlag = true; 6544 //import std.stdio; writeln("exit heard"); 6545 break; 6546 } else { 6547 foreach(listener; lcm.listeners) { 6548 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6549 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6550 // this try/catch is because it is set to non-blocking mode 6551 // and Phobos' stupid api throws an exception instead of returning 6552 // if it would block. Why would it block? because a forked process 6553 // might have beat us to it, but the wakeup event thundered our herds. 6554 try 6555 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6556 catch(SocketAcceptException e) { continue outer; } 6557 6558 cloexec(sn); 6559 if(lcm.tcp) { 6560 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6561 // on the socket because we do some buffering internally. I think this helps, 6562 // certainly does for small requests, and I think it does for larger ones too 6563 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6564 6565 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6566 } 6567 6568 dg(sn); 6569 continue outer; 6570 } else { 6571 // writeln(events[idx].data.ptr); 6572 } 6573 } 6574 6575 if(cast(size_t) events[idx].data.ptr < 1024) { 6576 throw arsd.core.ArsdException!"this doesn't look like a fiber pointer... "(cast(size_t) events[idx].data.ptr); 6577 } 6578 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6579 fiber.proceed(); 6580 } 6581 } 6582 } 6583 } 6584 6585 ListeningConnectionManager lcm; 6586 CMT dg; 6587 int myThreadNumber; 6588 } 6589 6590 6591 /* Done with network helper */ 6592 6593 /* Helpers for doing temporary files. Used both here and in web.d */ 6594 6595 version(Windows) { 6596 import core.sys.windows.windows; 6597 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6598 alias GetTempPathW GetTempPath; 6599 } 6600 6601 version(Posix) { 6602 static import linux = core.sys.posix.unistd; 6603 } 6604 6605 string getTempDirectory() { 6606 string path; 6607 version(Windows) { 6608 wchar[1024] buffer; 6609 auto len = GetTempPath(1024, buffer.ptr); 6610 if(len == 0) 6611 throw new Exception("couldn't find a temporary path"); 6612 6613 auto b = buffer[0 .. len]; 6614 6615 path = to!string(b); 6616 } else 6617 path = "/tmp/"; 6618 6619 return path; 6620 } 6621 6622 6623 // I like std.date. These functions help keep my old code and data working with phobos changing. 6624 6625 long sysTimeToDTime(in SysTime sysTime) { 6626 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6627 } 6628 6629 long dateTimeToDTime(in DateTime dt) { 6630 return sysTimeToDTime(cast(SysTime) dt); 6631 } 6632 6633 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6634 return sysTimeToDTime(Clock.currTime(UTC())); 6635 } 6636 6637 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6638 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6639 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6640 return SysTime(hnsecs, tz); 6641 } 6642 6643 6644 6645 // this is a helper to read HTTP transfer-encoding: chunked responses 6646 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6647 immutable(ubyte)[] ret; 6648 6649 another_chunk: 6650 // If here, we are at the beginning of a chunk. 6651 auto a = ir.front(); 6652 int chunkSize; 6653 int loc = locationOf(a, "\r\n"); 6654 while(loc == -1) { 6655 ir.popFront(); 6656 a = ir.front(); 6657 loc = locationOf(a, "\r\n"); 6658 } 6659 6660 string hex; 6661 hex = ""; 6662 for(int i = 0; i < loc; i++) { 6663 char c = a[i]; 6664 if(c >= 'A' && c <= 'Z') 6665 c += 0x20; 6666 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6667 hex ~= c; 6668 } else { 6669 break; 6670 } 6671 } 6672 6673 assert(hex.length); 6674 6675 int power = 1; 6676 int size = 0; 6677 foreach(cc1; retro(hex)) { 6678 dchar cc = cc1; 6679 if(cc >= 'a' && cc <= 'z') 6680 cc -= 0x20; 6681 int val = 0; 6682 if(cc >= '0' && cc <= '9') 6683 val = cc - '0'; 6684 else 6685 val = cc - 'A' + 10; 6686 6687 size += power * val; 6688 power *= 16; 6689 } 6690 6691 chunkSize = size; 6692 assert(size >= 0); 6693 6694 if(loc + 2 > a.length) { 6695 ir.popFront(0, a.length + loc + 2); 6696 a = ir.front(); 6697 } 6698 6699 a = ir.consume(loc + 2); 6700 6701 if(chunkSize == 0) { // we're done with the response 6702 // if we got here, will change must be true.... 6703 more_footers: 6704 loc = locationOf(a, "\r\n"); 6705 if(loc == -1) { 6706 ir.popFront(); 6707 a = ir.front; 6708 goto more_footers; 6709 } else { 6710 assert(loc == 0); 6711 ir.consume(loc + 2); 6712 goto finish; 6713 } 6714 } else { 6715 // if we got here, will change must be true.... 6716 if(a.length < chunkSize + 2) { 6717 ir.popFront(0, chunkSize + 2); 6718 a = ir.front(); 6719 } 6720 6721 ret ~= (a[0..chunkSize]); 6722 6723 if(!(a.length > chunkSize + 2)) { 6724 ir.popFront(0, chunkSize + 2); 6725 a = ir.front(); 6726 } 6727 assert(a[chunkSize] == 13); 6728 assert(a[chunkSize+1] == 10); 6729 a = ir.consume(chunkSize + 2); 6730 chunkSize = 0; 6731 goto another_chunk; 6732 } 6733 6734 finish: 6735 return ret; 6736 } 6737 6738 // I want to be able to get data from multiple sources the same way... 6739 interface ByChunkRange { 6740 bool empty(); 6741 void popFront(); 6742 const(ubyte)[] front(); 6743 } 6744 6745 ByChunkRange byChunk(const(ubyte)[] data) { 6746 return new class ByChunkRange { 6747 override bool empty() { 6748 return !data.length; 6749 } 6750 6751 override void popFront() { 6752 if(data.length > 4096) 6753 data = data[4096 .. $]; 6754 else 6755 data = null; 6756 } 6757 6758 override const(ubyte)[] front() { 6759 return data[0 .. $ > 4096 ? 4096 : $]; 6760 } 6761 }; 6762 } 6763 6764 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6765 const(ubyte)[] f; 6766 6767 f = ir.front; 6768 if(f.length > atMost) 6769 f = f[0 .. atMost]; 6770 6771 return new class ByChunkRange { 6772 override bool empty() { 6773 return atMost == 0; 6774 } 6775 6776 override const(ubyte)[] front() { 6777 return f; 6778 } 6779 6780 override void popFront() { 6781 ir.consume(f.length); 6782 atMost -= f.length; 6783 auto a = ir.front(); 6784 6785 if(a.length <= atMost) { 6786 f = a; 6787 atMost -= a.length; 6788 a = ir.consume(a.length); 6789 if(atMost != 0) 6790 ir.popFront(); 6791 if(f.length == 0) { 6792 f = ir.front(); 6793 } 6794 } else { 6795 // we actually have *more* here than we need.... 6796 f = a[0..atMost]; 6797 atMost = 0; 6798 ir.consume(atMost); 6799 } 6800 } 6801 }; 6802 } 6803 6804 version(cgi_with_websocket) { 6805 // http://tools.ietf.org/html/rfc6455 6806 6807 /++ 6808 WEBSOCKET SUPPORT: 6809 6810 Full example: 6811 --- 6812 import arsd.cgi; 6813 6814 void websocketEcho(Cgi cgi) { 6815 if(cgi.websocketRequested()) { 6816 if(cgi.origin != "http://arsdnet.net") 6817 throw new Exception("bad origin"); 6818 auto websocket = cgi.acceptWebsocket(); 6819 6820 websocket.send("hello"); 6821 websocket.send(" world!"); 6822 6823 auto msg = websocket.recv(); 6824 while(msg.opcode != WebSocketOpcode.close) { 6825 if(msg.opcode == WebSocketOpcode.text) { 6826 websocket.send(msg.textData); 6827 } else if(msg.opcode == WebSocketOpcode.binary) { 6828 websocket.send(msg.data); 6829 } 6830 6831 msg = websocket.recv(); 6832 } 6833 6834 websocket.close(); 6835 } else { 6836 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6837 } 6838 } 6839 6840 mixin GenericMain!websocketEcho; 6841 --- 6842 +/ 6843 6844 class WebSocket { 6845 Cgi cgi; 6846 6847 private this(Cgi cgi) { 6848 this.cgi = cgi; 6849 6850 Socket socket = cgi.idlol.source; 6851 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6852 } 6853 6854 // returns true if data available, false if it timed out 6855 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6856 if(!waitForNextMessageWouldBlock()) 6857 return true; 6858 if(isDataPending(timeout)) 6859 return true; // this is kinda a lie. 6860 6861 return false; 6862 } 6863 6864 public bool lowLevelReceive() { 6865 auto bfr = cgi.idlol; 6866 top: 6867 auto got = bfr.front; 6868 if(got.length) { 6869 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6870 receiveBuffer.length += receiveBufferUsedLength + got.length; 6871 6872 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6873 receiveBufferUsedLength += got.length; 6874 bfr.consume(got.length); 6875 6876 return true; 6877 } 6878 6879 if(bfr.sourceClosed) 6880 return false; 6881 6882 bfr.popFront(0); 6883 if(bfr.sourceClosed) 6884 return false; 6885 goto top; 6886 } 6887 6888 6889 bool isDataPending(Duration timeout = 0.seconds) { 6890 Socket socket = cgi.idlol.source; 6891 6892 auto check = new SocketSet(); 6893 check.add(socket); 6894 6895 auto got = Socket.select(check, null, null, timeout); 6896 if(got > 0) 6897 return true; 6898 return false; 6899 } 6900 6901 // note: this blocks 6902 WebSocketFrame recv() { 6903 return waitForNextMessage(); 6904 } 6905 6906 6907 6908 6909 private void llclose() { 6910 cgi.close(); 6911 } 6912 6913 private void llsend(ubyte[] data) { 6914 cgi.write(data); 6915 cgi.flush(); 6916 } 6917 6918 void unregisterActiveSocket(WebSocket) {} 6919 6920 /* copy/paste section { */ 6921 6922 private int readyState_; 6923 private ubyte[] receiveBuffer; 6924 private size_t receiveBufferUsedLength; 6925 6926 private Config config; 6927 6928 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6929 enum OPEN = 1; /// The connection is open and ready to communicate. 6930 enum CLOSING = 2; /// The connection is in the process of closing. 6931 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6932 6933 /++ 6934 6935 +/ 6936 /// Group: foundational 6937 static struct Config { 6938 /++ 6939 These control the size of the receive buffer. 6940 6941 It starts at the initial size, will temporarily 6942 balloon up to the maximum size, and will reuse 6943 a buffer up to the likely size. 6944 6945 Anything larger than the maximum size will cause 6946 the connection to be aborted and an exception thrown. 6947 This is to protect you against a peer trying to 6948 exhaust your memory, while keeping the user-level 6949 processing simple. 6950 +/ 6951 size_t initialReceiveBufferSize = 4096; 6952 size_t likelyReceiveBufferSize = 4096; /// ditto 6953 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6954 6955 /++ 6956 Maximum combined size of a message. 6957 +/ 6958 size_t maximumMessageSize = 10 * 1024 * 1024; 6959 6960 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6961 string origin; /// Origin URL to send with the handshake, if desired. 6962 string protocol; /// the protocol header, if desired. 6963 6964 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6965 } 6966 6967 /++ 6968 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6969 +/ 6970 int readyState() { 6971 return readyState_; 6972 } 6973 6974 /++ 6975 Closes the connection, sending a graceful teardown message to the other side. 6976 +/ 6977 /// Group: foundational 6978 void close(int code = 0, string reason = null) 6979 //in (reason.length < 123) 6980 in { assert(reason.length < 123); } do 6981 { 6982 if(readyState_ != OPEN) 6983 return; // it cool, we done 6984 WebSocketFrame wss; 6985 wss.fin = true; 6986 wss.opcode = WebSocketOpcode.close; 6987 wss.data = cast(ubyte[]) reason.dup; 6988 wss.send(&llsend); 6989 6990 readyState_ = CLOSING; 6991 6992 llclose(); 6993 } 6994 6995 /++ 6996 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6997 +/ 6998 /// Group: foundational 6999 void ping() { 7000 WebSocketFrame wss; 7001 wss.fin = true; 7002 wss.opcode = WebSocketOpcode.ping; 7003 wss.send(&llsend); 7004 } 7005 7006 // automatically handled.... 7007 void pong() { 7008 WebSocketFrame wss; 7009 wss.fin = true; 7010 wss.opcode = WebSocketOpcode.pong; 7011 wss.send(&llsend); 7012 } 7013 7014 /++ 7015 Sends a text message through the websocket. 7016 +/ 7017 /// Group: foundational 7018 void send(in char[] textData) { 7019 WebSocketFrame wss; 7020 wss.fin = true; 7021 wss.opcode = WebSocketOpcode.text; 7022 wss.data = cast(ubyte[]) textData.dup; 7023 wss.send(&llsend); 7024 } 7025 7026 /++ 7027 Sends a binary message through the websocket. 7028 +/ 7029 /// Group: foundational 7030 void send(in ubyte[] binaryData) { 7031 WebSocketFrame wss; 7032 wss.fin = true; 7033 wss.opcode = WebSocketOpcode.binary; 7034 wss.data = cast(ubyte[]) binaryData.dup; 7035 wss.send(&llsend); 7036 } 7037 7038 /++ 7039 Waits for and returns the next complete message on the socket. 7040 7041 Note that the onmessage function is still called, right before 7042 this returns. 7043 +/ 7044 /// Group: blocking_api 7045 public WebSocketFrame waitForNextMessage() { 7046 do { 7047 auto m = processOnce(); 7048 if(m.populated) 7049 return m; 7050 } while(lowLevelReceive()); 7051 7052 throw new ConnectionClosedException("Websocket receive timed out"); 7053 //return WebSocketFrame.init; // FIXME? maybe. 7054 } 7055 7056 /++ 7057 Tells if [waitForNextMessage] would block. 7058 +/ 7059 /// Group: blocking_api 7060 public bool waitForNextMessageWouldBlock() { 7061 checkAgain: 7062 if(isMessageBuffered()) 7063 return false; 7064 if(!isDataPending()) 7065 return true; 7066 while(isDataPending()) 7067 lowLevelReceive(); 7068 goto checkAgain; 7069 } 7070 7071 /++ 7072 Is there a message in the buffer already? 7073 If `true`, [waitForNextMessage] is guaranteed to return immediately. 7074 If `false`, check [isDataPending] as the next step. 7075 +/ 7076 /// Group: blocking_api 7077 public bool isMessageBuffered() { 7078 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 7079 auto s = d; 7080 if(d.length) { 7081 auto orig = d; 7082 auto m = WebSocketFrame.read(d); 7083 // that's how it indicates that it needs more data 7084 if(d !is orig) 7085 return true; 7086 } 7087 7088 return false; 7089 } 7090 7091 private ubyte continuingType; 7092 private ubyte[] continuingData; 7093 //private size_t continuingDataLength; 7094 7095 private WebSocketFrame processOnce() { 7096 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 7097 auto s = d; 7098 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 7099 WebSocketFrame m; 7100 if(d.length) { 7101 auto orig = d; 7102 m = WebSocketFrame.read(d); 7103 // that's how it indicates that it needs more data 7104 if(d is orig) 7105 return WebSocketFrame.init; 7106 m.unmaskInPlace(); 7107 switch(m.opcode) { 7108 case WebSocketOpcode.continuation: 7109 if(continuingData.length + m.data.length > config.maximumMessageSize) 7110 throw new Exception("message size exceeded"); 7111 7112 continuingData ~= m.data; 7113 if(m.fin) { 7114 if(ontextmessage) 7115 ontextmessage(cast(char[]) continuingData); 7116 if(onbinarymessage) 7117 onbinarymessage(continuingData); 7118 7119 continuingData = null; 7120 } 7121 break; 7122 case WebSocketOpcode.text: 7123 if(m.fin) { 7124 if(ontextmessage) 7125 ontextmessage(m.textData); 7126 } else { 7127 continuingType = m.opcode; 7128 //continuingDataLength = 0; 7129 continuingData = null; 7130 continuingData ~= m.data; 7131 } 7132 break; 7133 case WebSocketOpcode.binary: 7134 if(m.fin) { 7135 if(onbinarymessage) 7136 onbinarymessage(m.data); 7137 } else { 7138 continuingType = m.opcode; 7139 //continuingDataLength = 0; 7140 continuingData = null; 7141 continuingData ~= m.data; 7142 } 7143 break; 7144 case WebSocketOpcode.close: 7145 readyState_ = CLOSED; 7146 if(onclose) 7147 onclose(); 7148 7149 unregisterActiveSocket(this); 7150 break; 7151 case WebSocketOpcode.ping: 7152 pong(); 7153 break; 7154 case WebSocketOpcode.pong: 7155 // just really references it is still alive, nbd. 7156 break; 7157 default: // ignore though i could and perhaps should throw too 7158 } 7159 } 7160 7161 // the recv thing can be invalidated so gotta copy it over ugh 7162 if(d.length) { 7163 m.data = m.data.dup(); 7164 } 7165 7166 import core.stdc.string; 7167 memmove(receiveBuffer.ptr, d.ptr, d.length); 7168 receiveBufferUsedLength = d.length; 7169 7170 return m; 7171 } 7172 7173 private void autoprocess() { 7174 // FIXME 7175 do { 7176 processOnce(); 7177 } while(lowLevelReceive()); 7178 } 7179 7180 7181 void delegate() onclose; /// 7182 void delegate() onerror; /// 7183 void delegate(in char[]) ontextmessage; /// 7184 void delegate(in ubyte[]) onbinarymessage; /// 7185 void delegate() onopen; /// 7186 7187 /++ 7188 7189 +/ 7190 /// Group: browser_api 7191 void onmessage(void delegate(in char[]) dg) { 7192 ontextmessage = dg; 7193 } 7194 7195 /// ditto 7196 void onmessage(void delegate(in ubyte[]) dg) { 7197 onbinarymessage = dg; 7198 } 7199 7200 /* } end copy/paste */ 7201 7202 7203 } 7204 7205 /++ 7206 Returns true if the request headers are asking for a websocket upgrade. 7207 7208 If this returns true, and you want to accept it, call [acceptWebsocket]. 7209 +/ 7210 bool websocketRequested(Cgi cgi) { 7211 return 7212 "sec-websocket-key" in cgi.requestHeaders 7213 && 7214 "connection" in cgi.requestHeaders && 7215 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 7216 && 7217 "upgrade" in cgi.requestHeaders && 7218 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 7219 ; 7220 } 7221 7222 /++ 7223 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 7224 +/ 7225 WebSocket acceptWebsocket(Cgi cgi) { 7226 assert(!cgi.closed); 7227 assert(!cgi.outputtedResponseData); 7228 cgi.setResponseStatus("101 Switching Protocols"); 7229 cgi.header("Upgrade: WebSocket"); 7230 cgi.header("Connection: upgrade"); 7231 7232 string key = cgi.requestHeaders["sec-websocket-key"]; 7233 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 7234 7235 import std.digest.sha; 7236 auto hash = sha1Of(key); 7237 auto accept = Base64.encode(hash); 7238 7239 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 7240 7241 cgi.websocketMode = true; 7242 cgi.write(""); 7243 7244 cgi.flush(); 7245 7246 return new WebSocket(cgi); 7247 } 7248 7249 // FIXME get websocket to work on other modes, not just embedded_httpd 7250 7251 /* copy/paste in http2.d { */ 7252 enum WebSocketOpcode : ubyte { 7253 continuation = 0, 7254 text = 1, 7255 binary = 2, 7256 // 3, 4, 5, 6, 7 RESERVED 7257 close = 8, 7258 ping = 9, 7259 pong = 10, 7260 // 11,12,13,14,15 RESERVED 7261 } 7262 7263 public struct WebSocketFrame { 7264 private bool populated; 7265 bool fin; 7266 bool rsv1; 7267 bool rsv2; 7268 bool rsv3; 7269 WebSocketOpcode opcode; // 4 bits 7270 bool masked; 7271 ubyte lengthIndicator; // don't set this when building one to send 7272 ulong realLength; // don't use when sending 7273 ubyte[4] maskingKey; // don't set this when sending 7274 ubyte[] data; 7275 7276 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7277 WebSocketFrame msg; 7278 msg.fin = true; 7279 msg.opcode = opcode; 7280 msg.data = cast(ubyte[]) data.dup; 7281 7282 return msg; 7283 } 7284 7285 private void send(scope void delegate(ubyte[]) llsend) { 7286 ubyte[64] headerScratch; 7287 int headerScratchPos = 0; 7288 7289 realLength = data.length; 7290 7291 { 7292 ubyte b1; 7293 b1 |= cast(ubyte) opcode; 7294 b1 |= rsv3 ? (1 << 4) : 0; 7295 b1 |= rsv2 ? (1 << 5) : 0; 7296 b1 |= rsv1 ? (1 << 6) : 0; 7297 b1 |= fin ? (1 << 7) : 0; 7298 7299 headerScratch[0] = b1; 7300 headerScratchPos++; 7301 } 7302 7303 { 7304 headerScratchPos++; // we'll set header[1] at the end of this 7305 auto rlc = realLength; 7306 ubyte b2; 7307 b2 |= masked ? (1 << 7) : 0; 7308 7309 assert(headerScratchPos == 2); 7310 7311 if(realLength > 65535) { 7312 // use 64 bit length 7313 b2 |= 0x7f; 7314 7315 // FIXME: double check endinaness 7316 foreach(i; 0 .. 8) { 7317 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7318 rlc >>>= 8; 7319 } 7320 7321 headerScratchPos += 8; 7322 } else if(realLength > 125) { 7323 // use 16 bit length 7324 b2 |= 0x7e; 7325 7326 // FIXME: double check endinaness 7327 foreach(i; 0 .. 2) { 7328 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7329 rlc >>>= 8; 7330 } 7331 7332 headerScratchPos += 2; 7333 } else { 7334 // use 7 bit length 7335 b2 |= realLength & 0b_0111_1111; 7336 } 7337 7338 headerScratch[1] = b2; 7339 } 7340 7341 //assert(!masked, "masking key not properly implemented"); 7342 if(masked) { 7343 // FIXME: randomize this 7344 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7345 headerScratchPos += 4; 7346 7347 // we'll just mask it in place... 7348 int keyIdx = 0; 7349 foreach(i; 0 .. data.length) { 7350 data[i] = data[i] ^ maskingKey[keyIdx]; 7351 if(keyIdx == 3) 7352 keyIdx = 0; 7353 else 7354 keyIdx++; 7355 } 7356 } 7357 7358 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7359 llsend(headerScratch[0 .. headerScratchPos]); 7360 llsend(data); 7361 } 7362 7363 static WebSocketFrame read(ref ubyte[] d) { 7364 WebSocketFrame msg; 7365 7366 auto orig = d; 7367 7368 WebSocketFrame needsMoreData() { 7369 d = orig; 7370 return WebSocketFrame.init; 7371 } 7372 7373 if(d.length < 2) 7374 return needsMoreData(); 7375 7376 ubyte b = d[0]; 7377 7378 msg.populated = true; 7379 7380 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7381 b >>= 4; 7382 msg.rsv3 = b & 0x01; 7383 b >>= 1; 7384 msg.rsv2 = b & 0x01; 7385 b >>= 1; 7386 msg.rsv1 = b & 0x01; 7387 b >>= 1; 7388 msg.fin = b & 0x01; 7389 7390 b = d[1]; 7391 msg.masked = (b & 0b1000_0000) ? true : false; 7392 msg.lengthIndicator = b & 0b0111_1111; 7393 7394 d = d[2 .. $]; 7395 7396 if(msg.lengthIndicator == 0x7e) { 7397 // 16 bit length 7398 msg.realLength = 0; 7399 7400 if(d.length < 2) return needsMoreData(); 7401 7402 foreach(i; 0 .. 2) { 7403 msg.realLength |= d[0] << ((1-i) * 8); 7404 d = d[1 .. $]; 7405 } 7406 } else if(msg.lengthIndicator == 0x7f) { 7407 // 64 bit length 7408 msg.realLength = 0; 7409 7410 if(d.length < 8) return needsMoreData(); 7411 7412 foreach(i; 0 .. 8) { 7413 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7414 d = d[1 .. $]; 7415 } 7416 } else { 7417 // 7 bit length 7418 msg.realLength = msg.lengthIndicator; 7419 } 7420 7421 if(msg.masked) { 7422 7423 if(d.length < 4) return needsMoreData(); 7424 7425 msg.maskingKey = d[0 .. 4]; 7426 d = d[4 .. $]; 7427 } 7428 7429 if(msg.realLength > d.length) { 7430 return needsMoreData(); 7431 } 7432 7433 msg.data = d[0 .. cast(size_t) msg.realLength]; 7434 d = d[cast(size_t) msg.realLength .. $]; 7435 7436 return msg; 7437 } 7438 7439 void unmaskInPlace() { 7440 if(this.masked) { 7441 int keyIdx = 0; 7442 foreach(i; 0 .. this.data.length) { 7443 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7444 if(keyIdx == 3) 7445 keyIdx = 0; 7446 else 7447 keyIdx++; 7448 } 7449 } 7450 } 7451 7452 char[] textData() { 7453 return cast(char[]) data; 7454 } 7455 } 7456 /* } */ 7457 } 7458 7459 7460 version(Windows) 7461 { 7462 version(CRuntime_DigitalMars) 7463 { 7464 extern(C) int setmode(int, int) nothrow @nogc; 7465 } 7466 else version(CRuntime_Microsoft) 7467 { 7468 extern(C) int _setmode(int, int) nothrow @nogc; 7469 alias setmode = _setmode; 7470 } 7471 else static assert(0); 7472 } 7473 7474 version(Posix) { 7475 import core.sys.posix.unistd; 7476 version(CRuntime_Musl) {} else { 7477 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7478 } 7479 } 7480 7481 7482 // FIXME: these aren't quite public yet. 7483 //private: 7484 7485 // template for laziness 7486 void startAddonServer()(string arg) { 7487 version(OSX) { 7488 assert(0, "Not implemented"); 7489 } else version(linux) { 7490 import core.sys.posix.unistd; 7491 pid_t pid; 7492 const(char)*[16] args; 7493 args[0] = "ARSD_CGI_ADDON_SERVER"; 7494 args[1] = arg.ptr; 7495 posix_spawn(&pid, "/proc/self/exe", 7496 null, 7497 null, 7498 args.ptr, 7499 null // env 7500 ); 7501 } else version(Windows) { 7502 wchar[2048] filename; 7503 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7504 if(len == 0 || len == filename.length) 7505 throw new Exception("could not get process name to start helper server"); 7506 7507 STARTUPINFOW startupInfo; 7508 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7509 PROCESS_INFORMATION processInfo; 7510 7511 import std.utf; 7512 7513 // I *MIGHT* need to run it as a new job or a service... 7514 auto ret = CreateProcessW( 7515 filename.ptr, 7516 toUTF16z(arg), 7517 null, // process attributes 7518 null, // thread attributes 7519 false, // inherit handles 7520 0, // creation flags 7521 null, // environment 7522 null, // working directory 7523 &startupInfo, 7524 &processInfo 7525 ); 7526 7527 if(!ret) 7528 throw new Exception("create process failed"); 7529 7530 // when done with those, if we set them 7531 /* 7532 CloseHandle(hStdInput); 7533 CloseHandle(hStdOutput); 7534 CloseHandle(hStdError); 7535 */ 7536 7537 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7538 } 7539 7540 // template for laziness 7541 /* 7542 The websocket server is a single-process, single-thread, event 7543 I/O thing. It is passed websockets from other CGI processes 7544 and is then responsible for handling their messages and responses. 7545 Note that the CGI process is responsible for websocket setup, 7546 including authentication, etc. 7547 7548 It also gets data sent to it by other processes and is responsible 7549 for distributing that, as necessary. 7550 */ 7551 void runWebsocketServer()() { 7552 assert(0, "not implemented"); 7553 } 7554 7555 void sendToWebsocketServer(WebSocket ws, string group) { 7556 assert(0, "not implemented"); 7557 } 7558 7559 void sendToWebsocketServer(string content, string group) { 7560 assert(0, "not implemented"); 7561 } 7562 7563 7564 void runEventServer()() { 7565 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7566 } 7567 7568 void runTimerServer()() { 7569 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7570 } 7571 7572 version(Posix) { 7573 alias LocalServerConnectionHandle = int; 7574 alias CgiConnectionHandle = int; 7575 alias SocketConnectionHandle = int; 7576 7577 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7578 } else version(Windows) { 7579 alias LocalServerConnectionHandle = HANDLE; 7580 version(embedded_httpd_threads) { 7581 alias CgiConnectionHandle = SOCKET; 7582 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7583 } else version(fastcgi) { 7584 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7585 enum INVALID_CGI_CONNECTION_HANDLE = null; 7586 } else version(scgi) { 7587 alias CgiConnectionHandle = SOCKET; 7588 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7589 } else { /* version(plain_cgi) */ 7590 alias CgiConnectionHandle = HANDLE; 7591 enum INVALID_CGI_CONNECTION_HANDLE = null; 7592 } 7593 alias SocketConnectionHandle = SOCKET; 7594 } 7595 7596 version(with_addon_servers_connections) 7597 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7598 version(Posix) { 7599 import core.sys.posix.unistd; 7600 import core.sys.posix.sys.un; 7601 7602 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7603 if(sock == -1) 7604 throw new Exception("socket " ~ to!string(errno)); 7605 7606 scope(failure) 7607 close(sock); 7608 7609 cloexec(sock); 7610 7611 // add-on server processes are assumed to be local, and thus will 7612 // use unix domain sockets. Besides, I want to pass sockets to them, 7613 // so it basically must be local (except for the session server, but meh). 7614 sockaddr_un addr; 7615 addr.sun_family = AF_UNIX; 7616 version(linux) { 7617 // on linux, we will use the abstract namespace 7618 addr.sun_path[0] = 0; 7619 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7620 } else { 7621 // but otherwise, just use a file cuz we must. 7622 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7623 } 7624 7625 bool alreadyTried; 7626 7627 try_again: 7628 7629 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7630 if(!alreadyTried && errno == ECONNREFUSED) { 7631 // try auto-spawning the server, then attempt connection again 7632 startAddonServer(arg); 7633 import core.thread; 7634 Thread.sleep(50.msecs); 7635 alreadyTried = true; 7636 goto try_again; 7637 } else 7638 throw new Exception("connect " ~ to!string(errno)); 7639 } 7640 7641 return sock; 7642 } else version(Windows) { 7643 return null; // FIXME 7644 } 7645 } 7646 7647 version(with_addon_servers_connections) 7648 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7649 version(Posix) { 7650 import core.sys.posix.unistd; 7651 close(handle); 7652 } else version(Windows) 7653 CloseHandle(handle); 7654 } 7655 7656 void runSessionServer()() { 7657 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7658 } 7659 7660 import core.stdc.errno; 7661 7662 struct IoOp { 7663 @disable this(); 7664 @disable this(this); 7665 7666 /* 7667 So we want to be able to eventually handle generic sockets too. 7668 */ 7669 7670 enum Read = 1; 7671 enum Write = 2; 7672 enum Accept = 3; 7673 enum ReadSocketHandle = 4; 7674 7675 // Your handler may be called in a different thread than the one that initiated the IO request! 7676 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7677 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7678 private void delegate(IoOp*) closeHandler; 7679 private void delegate(IoOp*) completeHandler; 7680 private int internalFd; 7681 private int operation; 7682 private int bufferLengthAllocated; 7683 private int bufferLengthUsed; 7684 private ubyte[1] internalBuffer; // it can be overallocated! 7685 7686 ubyte[] allocatedBuffer() return { 7687 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7688 } 7689 7690 ubyte[] usedBuffer() return { 7691 return allocatedBuffer[0 .. bufferLengthUsed]; 7692 } 7693 7694 void reset() { 7695 bufferLengthUsed = 0; 7696 } 7697 7698 int fd() { 7699 return internalFd; 7700 } 7701 } 7702 7703 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7704 import core.stdc.stdlib; 7705 7706 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7707 if(ptr is null) 7708 assert(0); // out of memory! 7709 7710 auto op = cast(IoOp*) ptr; 7711 7712 op.handler = handler; 7713 op.internalFd = fd; 7714 op.operation = operation; 7715 op.bufferLengthAllocated = bufferSize; 7716 op.bufferLengthUsed = 0; 7717 7718 import core.memory; 7719 7720 GC.addRoot(ptr); 7721 7722 return op; 7723 } 7724 7725 void freeIoOp(ref IoOp* ptr) { 7726 7727 import core.memory; 7728 GC.removeRoot(ptr); 7729 7730 import core.stdc.stdlib; 7731 free(ptr); 7732 ptr = null; 7733 } 7734 7735 version(Posix) 7736 version(with_addon_servers_connections) 7737 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7738 7739 //import std.stdio : writeln; writeln(cast(string) data); 7740 7741 import core.sys.posix.unistd; 7742 7743 auto ret = write(connection, data.ptr, data.length); 7744 if(ret != data.length) { 7745 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7746 // the file is closed, remove it 7747 eis.fileClosed(connection); 7748 } else 7749 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7750 } 7751 } 7752 version(Windows) 7753 version(with_addon_servers_connections) 7754 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7755 // FIXME 7756 } 7757 7758 bool isInvalidHandle(CgiConnectionHandle h) { 7759 return h == INVALID_CGI_CONNECTION_HANDLE; 7760 } 7761 7762 /+ 7763 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7764 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7765 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7766 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7767 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7768 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7769 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7770 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7771 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7772 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7773 7774 +/ 7775 7776 /++ 7777 You can customize your server by subclassing the appropriate server. Then, register your 7778 subclass at compile time with the [registerEventIoServer] template, or implement your own 7779 main function and call it yourself. 7780 7781 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7782 +/ 7783 version(with_addon_servers_connections) 7784 interface EventIoServer { 7785 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7786 void handleLocalConnectionClose(IoOp* op); 7787 void handleLocalConnectionComplete(IoOp* op); 7788 void wait_timeout(); 7789 void fileClosed(int fd); 7790 7791 void epoll_fd(int fd); 7792 } 7793 7794 // the sink should buffer it 7795 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7796 static if(is(T == struct)) { 7797 foreach(member; __traits(allMembers, T)) 7798 serialize(sink, __traits(getMember, t, member)); 7799 } else static if(is(T : int)) { 7800 // no need to think of endianness just because this is only used 7801 // for local, same-machine stuff anyway. thanks private lol 7802 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7803 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7804 // these are common enough to optimize 7805 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7806 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7807 sink(cast(ubyte[]) t[]); 7808 } else static if(is(T : A[], A)) { 7809 // generic array is less optimal but still prolly ok 7810 int len = cast(int) t.length; 7811 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7812 foreach(item; t) 7813 serialize(sink, item); 7814 } else static assert(0, T.stringof); 7815 } 7816 7817 // all may be stack buffers, so use cautio 7818 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7819 static if(is(T == struct)) { 7820 T t; 7821 foreach(member; __traits(allMembers, T)) 7822 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7823 dg(t); 7824 } else static if(is(T : int)) { 7825 // no need to think of endianness just because this is only used 7826 // for local, same-machine stuff anyway. thanks private lol 7827 T t; 7828 auto data = get(t.sizeof); 7829 t = (cast(T[]) data)[0]; 7830 dg(t); 7831 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7832 // these are common enough to optimize 7833 int len; 7834 auto data = get(len.sizeof); 7835 len = (cast(int[]) data)[0]; 7836 7837 /* 7838 typeof(T[0])[2000] stackBuffer; 7839 T buffer; 7840 7841 if(len < stackBuffer.length) 7842 buffer = stackBuffer[0 .. len]; 7843 else 7844 buffer = new T(len); 7845 7846 data = get(len * typeof(T[0]).sizeof); 7847 */ 7848 7849 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7850 7851 dg(t); 7852 } else static if(is(T == E[], E)) { 7853 T t; 7854 int len; 7855 auto data = get(len.sizeof); 7856 len = (cast(int[]) data)[0]; 7857 t.length = len; 7858 foreach(ref e; t) { 7859 deserialize!E(get, (ele) { e = ele; }); 7860 } 7861 dg(t); 7862 } else static assert(0, T.stringof); 7863 } 7864 7865 unittest { 7866 serialize((ubyte[] b) { 7867 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7868 }, 1); 7869 serialize((ubyte[] b) { 7870 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7871 }, 56674); 7872 ubyte[1000] buffer; 7873 int bufferPoint; 7874 void add(scope ubyte[] b) { 7875 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7876 bufferPoint += b.length; 7877 } 7878 ubyte[] get(int sz) { 7879 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7880 bufferPoint += sz; 7881 return b; 7882 } 7883 serialize(&add, "test here"); 7884 bufferPoint = 0; 7885 deserialize!string(&get, (t) { assert(t == "test here"); }); 7886 bufferPoint = 0; 7887 7888 struct Foo { 7889 int a; 7890 ubyte c; 7891 string d; 7892 } 7893 serialize(&add, Foo(403, 37, "amazing")); 7894 bufferPoint = 0; 7895 deserialize!Foo(&get, (t) { 7896 assert(t.a == 403); 7897 assert(t.c == 37); 7898 assert(t.d == "amazing"); 7899 }); 7900 bufferPoint = 0; 7901 } 7902 7903 /* 7904 Here's the way the RPC interface works: 7905 7906 You define the interface that lists the functions you can call on the remote process. 7907 The interface may also have static methods for convenience. These forward to a singleton 7908 instance of an auto-generated class, which actually sends the args over the pipe. 7909 7910 An impl class actually implements it. A receiving server deserializes down the pipe and 7911 calls methods on the class. 7912 7913 I went with the interface to get some nice compiler checking and documentation stuff. 7914 7915 I could have skipped the interface and just implemented it all from the server class definition 7916 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7917 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7918 7919 7920 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7921 7922 Realistically though the bodies would just be 7923 connection.call(this.mangleof, args...) sooooo. 7924 7925 FIXME: overloads aren't supported 7926 */ 7927 7928 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7929 interface SessionObject {} 7930 7931 private immutable void delegate(string[])[string] scheduledJobHandlers; 7932 private immutable void delegate(string[])[string] websocketServers; 7933 7934 version(with_breaking_cgi_features) 7935 mixin(q{ 7936 7937 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7938 static import std.traits; 7939 7940 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7941 static foreach(idx, member; __traits(derivedMembers, T)) { 7942 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7943 mixin( q{ 7944 std.traits.ReturnType!(__traits(getMember, T, member)) 7945 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7946 { 7947 SerializationBuffer buffer; 7948 auto i = cast(ushort) idx; 7949 serialize(&buffer.sink, i); 7950 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7951 foreach(param; params) 7952 serialize(&buffer.sink, param); 7953 7954 auto sendable = buffer.sendable; 7955 7956 version(Posix) {{ 7957 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7958 7959 if(ret == -1) { 7960 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7961 } else if(ret == 0) { 7962 throw new Exception("Connection to addon server lost"); 7963 } if(ret < sendable.length) 7964 throw new Exception("Send failed to send all"); 7965 assert(ret == sendable.length); 7966 }} // FIXME Windows impl 7967 7968 static if(!is(typeof(return) == void)) { 7969 // there is a return value; we need to wait for it too 7970 version(Posix) { 7971 ubyte[3000] revBuffer; 7972 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7973 auto got = revBuffer[0 .. ret]; 7974 7975 int dataLocation; 7976 ubyte[] grab(int sz) { 7977 auto dataLocation1 = dataLocation; 7978 dataLocation += sz; 7979 return got[dataLocation1 .. dataLocation]; 7980 } 7981 7982 typeof(return) retu; 7983 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7984 return retu; 7985 } else { 7986 // FIXME Windows impl 7987 return typeof(return).init; 7988 } 7989 7990 } 7991 }}); 7992 } 7993 7994 private static typeof(this) singletonInstance; 7995 private LocalServerConnectionHandle connectionHandle; 7996 7997 static typeof(this) connection() { 7998 if(singletonInstance is null) { 7999 singletonInstance = new typeof(this)(); 8000 singletonInstance.connect(); 8001 } 8002 return singletonInstance; 8003 } 8004 8005 void connect() { 8006 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 8007 } 8008 8009 void disconnect() { 8010 closeLocalServerConnection(connectionHandle); 8011 } 8012 } 8013 8014 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 8015 ushort calledIdx; 8016 string calledFunction; 8017 8018 int dataLocation; 8019 ubyte[] grab(int sz) { 8020 if(sz == 0) assert(0); 8021 auto d = data[dataLocation .. dataLocation + sz]; 8022 dataLocation += sz; 8023 return d; 8024 } 8025 8026 again: 8027 8028 deserialize!ushort(&grab, (a) { calledIdx = a; }); 8029 deserialize!string(&grab, (a) { calledFunction = a; }); 8030 8031 import std.traits; 8032 8033 sw: switch(calledIdx) { 8034 foreach(idx, memberName; __traits(derivedMembers, Interface)) 8035 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 8036 case idx: 8037 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 8038 8039 Parameters!(__traits(getMember, Interface, memberName)) params; 8040 foreach(ref param; params) 8041 deserialize!(typeof(param))(&grab, (a) { param = a; }); 8042 8043 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 8044 __traits(getMember, this_, memberName)(params); 8045 } else { 8046 auto ret = __traits(getMember, this_, memberName)(params); 8047 SerializationBuffer buffer; 8048 serialize(&buffer.sink, ret); 8049 8050 auto sendable = buffer.sendable; 8051 8052 version(Posix) { 8053 auto r = send(fd, sendable.ptr, sendable.length, 0); 8054 if(r == -1) { 8055 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 8056 } else if(r == 0) { 8057 throw new Exception("Connection to addon client lost"); 8058 } if(r < sendable.length) 8059 throw new Exception("Send failed to send all"); 8060 8061 } // FIXME Windows impl 8062 } 8063 break sw; 8064 } 8065 default: assert(0); 8066 } 8067 8068 if(dataLocation != data.length) 8069 goto again; 8070 } 8071 8072 8073 private struct SerializationBuffer { 8074 ubyte[2048] bufferBacking; 8075 int bufferLocation; 8076 void sink(scope ubyte[] data) { 8077 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 8078 bufferLocation += data.length; 8079 } 8080 8081 ubyte[] sendable() return { 8082 return bufferBacking[0 .. bufferLocation]; 8083 } 8084 } 8085 8086 /* 8087 FIXME: 8088 add a version command line arg 8089 version data in the library 8090 management gui as external program 8091 8092 at server with event_fd for each run 8093 use .mangleof in the at function name 8094 8095 i think the at server will have to: 8096 pipe args to the child 8097 collect child output for logging 8098 get child return value for logging 8099 8100 on windows timers work differently. idk how to best combine with the io stuff. 8101 8102 will have to have dump and restore too, so i can restart without losing stuff. 8103 */ 8104 8105 /++ 8106 A convenience object for talking to the [BasicDataServer] from a higher level. 8107 See: [Cgi.getSessionObject]. 8108 8109 You pass it a `Data` struct describing the data you want saved in the session. 8110 Then, this class will generate getter and setter properties that allow access 8111 to that data. 8112 8113 Note that each load and store will be done as-accessed; it doesn't front-load 8114 mutable data nor does it batch updates out of fear of read-modify-write race 8115 conditions. (In fact, right now it does this for everything, but in the future, 8116 I might batch load `immutable` members of the Data struct.) 8117 8118 At some point in the future, I might also let it do different backends, like 8119 a client-side cookie store too, but idk. 8120 8121 Note that the plain-old-data members of your `Data` struct are wrapped by this 8122 interface via a static foreach to make property functions. 8123 8124 See_Also: [MockSession] 8125 +/ 8126 interface Session(Data) : SessionObject { 8127 @property string sessionId() const; 8128 8129 /++ 8130 Starts a new session. Note that a session is also 8131 implicitly started as soon as you write data to it, 8132 so if you need to alter these parameters from their 8133 defaults, be sure to explicitly call this BEFORE doing 8134 any writes to session data. 8135 8136 Params: 8137 idleLifetime = How long, in seconds, the session 8138 should remain in memory when not being read from 8139 or written to. The default is one day. 8140 8141 NOT IMPLEMENTED 8142 8143 useExtendedLifetimeCookie = The session ID is always 8144 stored in a HTTP cookie, and by default, that cookie 8145 is discarded when the user closes their browser. 8146 8147 But if you set this to true, it will use a non-perishable 8148 cookie for the given idleLifetime. 8149 8150 NOT IMPLEMENTED 8151 +/ 8152 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 8153 8154 /++ 8155 Regenerates the session ID and updates the associated 8156 cookie. 8157 8158 This is also your chance to change immutable data 8159 (not yet implemented). 8160 +/ 8161 void regenerateId(); 8162 8163 /++ 8164 Terminates this session, deleting all saved data. 8165 +/ 8166 void terminate(); 8167 8168 /++ 8169 Plain-old-data members of your `Data` struct are wrapped here via 8170 the property getters and setters. 8171 8172 If the member is a non-string array, it returns a magical array proxy 8173 object which allows for atomic appends and replaces via overloaded operators. 8174 You can slice this to get a range representing a $(B const) view of the array. 8175 This is to protect you against read-modify-write race conditions. 8176 +/ 8177 static foreach(memberName; __traits(allMembers, Data)) 8178 static if(is(typeof(__traits(getMember, Data, memberName)))) 8179 mixin(q{ 8180 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 8181 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 8182 }); 8183 8184 } 8185 8186 /++ 8187 An implementation of [Session] that works on real cgi connections utilizing the 8188 [BasicDataServer]. 8189 8190 As opposed to a [MockSession] which is made for testing purposes. 8191 8192 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 8193 +/ 8194 class BasicDataServerSession(Data) : Session!Data { 8195 private Cgi cgi; 8196 private string sessionId_; 8197 8198 public @property string sessionId() const { 8199 return sessionId_; 8200 } 8201 8202 protected @property string sessionId(string s) { 8203 return this.sessionId_ = s; 8204 } 8205 8206 private this(Cgi cgi) { 8207 this.cgi = cgi; 8208 if(auto ptr = "sessionId" in cgi.cookies) 8209 sessionId = (*ptr).length ? *ptr : null; 8210 } 8211 8212 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 8213 assert(sessionId is null); 8214 8215 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 8216 8217 import std.random, std.conv; 8218 sessionId = to!string(uniform(1, long.max)); 8219 8220 BasicDataServer.connection.createSession(sessionId, idleLifetime); 8221 setCookie(); 8222 } 8223 8224 protected void setCookie() { 8225 cgi.setCookie( 8226 "sessionId", sessionId, 8227 0 /* expiration */, 8228 "/" /* path */, 8229 null /* domain */, 8230 true /* http only */, 8231 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 8232 } 8233 8234 void regenerateId() { 8235 if(sessionId is null) { 8236 start(); 8237 return; 8238 } 8239 import std.random, std.conv; 8240 auto oldSessionId = sessionId; 8241 sessionId = to!string(uniform(1, long.max)); 8242 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 8243 setCookie(); 8244 } 8245 8246 void terminate() { 8247 BasicDataServer.connection.destroySession(sessionId); 8248 sessionId = null; 8249 setCookie(); 8250 } 8251 8252 static foreach(memberName; __traits(allMembers, Data)) 8253 static if(is(typeof(__traits(getMember, Data, memberName)))) 8254 mixin(q{ 8255 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8256 if(sessionId is null) 8257 return typeof(return).init; 8258 8259 import std.traits; 8260 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8261 if(v.length == 0) 8262 return typeof(return).init; 8263 import std.conv; 8264 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8265 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8266 // basically. Assuming the session is POD this should be fine. 8267 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8268 } 8269 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8270 if(sessionId is null) 8271 start(); 8272 import std.conv; 8273 import std.traits; 8274 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8275 return value; 8276 } 8277 }); 8278 } 8279 8280 /++ 8281 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8282 Simply stores the data in its instance members. 8283 +/ 8284 class MockSession(Data) : Session!Data { 8285 pure { 8286 @property string sessionId() const { return "mock"; } 8287 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8288 void regenerateId() {} 8289 void terminate() {} 8290 8291 private Data store_; 8292 8293 static foreach(memberName; __traits(allMembers, Data)) 8294 static if(is(typeof(__traits(getMember, Data, memberName)))) 8295 mixin(q{ 8296 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8297 return __traits(getMember, store_, memberName); 8298 } 8299 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8300 return __traits(getMember, store_, memberName) = value; 8301 } 8302 }); 8303 } 8304 } 8305 8306 /++ 8307 Direct interface to the basic data add-on server. You can 8308 typically use [Cgi.getSessionObject] as a more convenient interface. 8309 +/ 8310 version(with_addon_servers_connections) 8311 interface BasicDataServer { 8312 /// 8313 void createSession(string sessionId, int lifetime); 8314 /// 8315 void renewSession(string sessionId, int lifetime); 8316 /// 8317 void destroySession(string sessionId); 8318 /// 8319 void renameSession(string oldSessionId, string newSessionId); 8320 8321 /// 8322 void setSessionData(string sessionId, string dataKey, string dataValue); 8323 /// 8324 string getSessionData(string sessionId, string dataKey); 8325 8326 /// 8327 static BasicDataServerConnection connection() { 8328 return BasicDataServerConnection.connection(); 8329 } 8330 } 8331 8332 version(with_addon_servers_connections) 8333 class BasicDataServerConnection : BasicDataServer { 8334 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8335 } 8336 8337 version(with_addon_servers) 8338 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8339 8340 void createSession(string sessionId, int lifetime) { 8341 sessions[sessionId.idup] = Session(lifetime); 8342 } 8343 void destroySession(string sessionId) { 8344 sessions.remove(sessionId); 8345 } 8346 void renewSession(string sessionId, int lifetime) { 8347 sessions[sessionId].lifetime = lifetime; 8348 } 8349 void renameSession(string oldSessionId, string newSessionId) { 8350 sessions[newSessionId.idup] = sessions[oldSessionId]; 8351 sessions.remove(oldSessionId); 8352 } 8353 void setSessionData(string sessionId, string dataKey, string dataValue) { 8354 if(sessionId !in sessions) 8355 createSession(sessionId, 3600); // FIXME? 8356 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8357 } 8358 string getSessionData(string sessionId, string dataKey) { 8359 if(auto session = sessionId in sessions) { 8360 if(auto data = dataKey in (*session).values) 8361 return *data; 8362 else 8363 return null; // no such data 8364 8365 } else { 8366 return null; // no session 8367 } 8368 } 8369 8370 8371 protected: 8372 8373 struct Session { 8374 int lifetime; 8375 8376 string[string] values; 8377 } 8378 8379 Session[string] sessions; 8380 8381 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8382 auto data = op.usedBuffer; 8383 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8384 return false; 8385 } 8386 8387 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8388 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8389 void wait_timeout() {} 8390 void fileClosed(int fd) {} // stateless so irrelevant 8391 void epoll_fd(int fd) {} 8392 } 8393 8394 /++ 8395 See [schedule] to make one of these. You then call one of the methods here to set it up: 8396 8397 --- 8398 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8399 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8400 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8401 --- 8402 +/ 8403 version(with_addon_servers_connections) 8404 struct ScheduledJobHelper { 8405 private string func; 8406 private string[] args; 8407 private bool consumed; 8408 8409 private this(string func, string[] args) { 8410 this.func = func; 8411 this.args = args; 8412 } 8413 8414 ~this() { 8415 assert(consumed); 8416 } 8417 8418 /++ 8419 Schedules the job to be run at the given time. 8420 +/ 8421 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8422 consumed = true; 8423 8424 auto conn = ScheduledJobServerConnection.connection; 8425 import std.file; 8426 auto st = SysTime(when, timezone); 8427 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8428 } 8429 8430 /++ 8431 Schedules the job to run at least after the specified delay. 8432 +/ 8433 void delay(Duration delay) { 8434 consumed = true; 8435 8436 auto conn = ScheduledJobServerConnection.connection; 8437 import std.file; 8438 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8439 } 8440 8441 /++ 8442 Runs the job in the background ASAP. 8443 8444 $(NOTE It may run in a background thread. Don't segfault!) 8445 +/ 8446 void asap() { 8447 consumed = true; 8448 8449 auto conn = ScheduledJobServerConnection.connection; 8450 import std.file; 8451 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8452 } 8453 8454 /+ 8455 /++ 8456 Schedules the job to recur on the given pattern. 8457 +/ 8458 void recur(string spec) { 8459 8460 } 8461 +/ 8462 } 8463 8464 /++ 8465 First step to schedule a job on the scheduled job server. 8466 8467 The scheduled job needs to be a top-level function that doesn't read any 8468 variables from outside its arguments because it may be run in a new process, 8469 without any context existing later. 8470 8471 You MUST set details on the returned object to actually do anything! 8472 +/ 8473 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8474 /// 8475 ScheduledJobHelper schedule(T args) { 8476 // this isn't meant to ever be called, but instead just to 8477 // get the compiler to type check the arguments passed for us 8478 auto sample = delegate() { 8479 fn(args); 8480 }; 8481 string[] sargs; 8482 foreach(arg; args) 8483 sargs ~= to!string(arg); 8484 return ScheduledJobHelper(fn.mangleof, sargs); 8485 } 8486 8487 shared static this() { 8488 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8489 import std.traits; 8490 Parameters!fn args; 8491 foreach(idx, ref arg; args) 8492 arg = to!(typeof(arg))(sargs[idx]); 8493 fn(args); 8494 }; 8495 } 8496 } 8497 8498 /// 8499 interface ScheduledJobServer { 8500 /// Use the [schedule] function for a higher-level interface. 8501 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8502 /// 8503 void cancelJob(int jobId); 8504 } 8505 8506 version(with_addon_servers_connections) 8507 class ScheduledJobServerConnection : ScheduledJobServer { 8508 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8509 } 8510 8511 version(with_addon_servers) 8512 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8513 // FIXME: we need to handle SIGCHLD in this somehow 8514 // whenIs is 0 for relative, 1 for absolute 8515 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8516 auto nj = nextJobId; 8517 nextJobId++; 8518 8519 version(linux) { 8520 import core.sys.linux.timerfd; 8521 import core.sys.linux.epoll; 8522 import core.sys.posix.unistd; 8523 8524 8525 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8526 if(fd == -1) 8527 throw new Exception("fd timer create failed"); 8528 8529 foreach(ref arg; args) 8530 arg = arg.idup; 8531 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8532 8533 itimerspec value; 8534 value.it_value.tv_sec = when; 8535 value.it_value.tv_nsec = 0; 8536 8537 value.it_interval.tv_sec = 0; 8538 value.it_interval.tv_nsec = 0; 8539 8540 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8541 throw new Exception("couldn't set fd timer"); 8542 8543 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8544 jobs.remove(nj); 8545 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8546 close(fd); 8547 8548 8549 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8550 8551 return true; 8552 }); 8553 scope(failure) 8554 freeIoOp(op); 8555 8556 epoll_event ev; 8557 ev.events = EPOLLIN | EPOLLET; 8558 ev.data.ptr = op; 8559 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8560 throw new Exception("epoll_ctl " ~ to!string(errno)); 8561 8562 jobs[nj] = job; 8563 return nj; 8564 } else assert(0); 8565 } 8566 8567 protected void cancelJob(int jobId) { 8568 version(linux) { 8569 auto job = jobId in jobs; 8570 if(job is null) 8571 return; 8572 8573 jobs.remove(jobId); 8574 8575 version(linux) { 8576 import core.sys.linux.timerfd; 8577 import core.sys.linux.epoll; 8578 import core.sys.posix.unistd; 8579 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8580 close(job.timerfd); 8581 } 8582 } 8583 jobs.remove(jobId); 8584 } 8585 8586 int nextJobId = 1; 8587 static struct Job { 8588 string executable; 8589 string func; 8590 string[] args; 8591 int timerfd; 8592 int id; 8593 } 8594 Job[int] jobs; 8595 8596 8597 // event io server methods below 8598 8599 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8600 auto data = op.usedBuffer; 8601 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8602 return false; 8603 } 8604 8605 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8606 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8607 void wait_timeout() {} 8608 void fileClosed(int fd) {} // stateless so irrelevant 8609 8610 int epoll_fd_; 8611 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8612 int epoll_fd() { return epoll_fd_; } 8613 } 8614 8615 /++ 8616 History: 8617 Added January 6, 2019 8618 +/ 8619 version(with_addon_servers_connections) 8620 interface EventSourceServer { 8621 /++ 8622 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8623 8624 See_Also: 8625 [sendEvent] 8626 8627 Bugs: 8628 Not implemented on Windows! 8629 8630 History: 8631 Officially stabilised on November 23, 2023 (dub v11.4). It actually worked pretty well in its original design. 8632 +/ 8633 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8634 /* 8635 If lastEventId is missing or empty, you just get new events as they come. 8636 8637 If it is set from something else, it sends all since then (that are still alive) 8638 down the pipe immediately. 8639 8640 The reason it can come from the header is that's what the standard defines for 8641 browser reconnects. The reason it can come from a query string is just convenience 8642 in catching up in a user-defined manner. 8643 8644 The reason the header overrides the query string is if the browser tries to reconnect, 8645 it will send the header AND the query (it reconnects to the same url), so we just 8646 want to do the restart thing. 8647 8648 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8649 */ 8650 string lastEventId = cgi.lastEventId; 8651 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8652 lastEventId = cgi.get["lastEventId"]; 8653 8654 cgi.setResponseContentType("text/event-stream"); 8655 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8656 cgi.flush(); 8657 8658 cgi.closed = true; 8659 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8660 scope(exit) 8661 closeLocalServerConnection(s); 8662 8663 version(fastcgi) 8664 throw new Exception("sending fcgi connections not supported"); 8665 else { 8666 auto fd = cgi.getOutputFileHandle(); 8667 if(isInvalidHandle(fd)) 8668 throw new Exception("bad fd from cgi!"); 8669 8670 EventSourceServerImplementation.SendableEventConnection sec; 8671 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8672 8673 version(Posix) { 8674 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8675 assert(res == sec.sizeof); 8676 } else version(Windows) { 8677 // FIXME 8678 } 8679 } 8680 } 8681 8682 /++ 8683 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8684 8685 Params: 8686 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8687 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8688 data = the event data. Available in JS as `event.data`. 8689 lifetime = the amount of time to keep this event for replaying on the event server. 8690 8691 Bugs: 8692 Not implemented on Windows! 8693 8694 History: 8695 Officially stabilised on November 23, 2023 (dub v11.4). It actually worked pretty well in its original design. 8696 +/ 8697 public static void sendEvent(string url, string event, string data, int lifetime) { 8698 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8699 scope(exit) 8700 closeLocalServerConnection(s); 8701 8702 EventSourceServerImplementation.SendableEvent sev; 8703 sev.populate(url, event, data, lifetime); 8704 8705 version(Posix) { 8706 auto ret = send(s, &sev, sev.sizeof, 0); 8707 assert(ret == sev.sizeof); 8708 } else version(Windows) { 8709 // FIXME 8710 } 8711 } 8712 8713 /++ 8714 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8715 8716 See_Also: [disconnect] 8717 +/ 8718 void connect(string url, string forwardUrl); 8719 8720 /++ 8721 Disconnects `forwardUrl` from `url` 8722 8723 See_Also: [connect] 8724 +/ 8725 void disconnect(string url, string forwardUrl); 8726 } 8727 8728 /// 8729 version(with_addon_servers) 8730 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8731 8732 protected: 8733 8734 void connect(string url, string forwardUrl) { 8735 pipes[url] ~= forwardUrl; 8736 } 8737 void disconnect(string url, string forwardUrl) { 8738 auto t = url in pipes; 8739 if(t is null) 8740 return; 8741 foreach(idx, n; (*t)) 8742 if(n == forwardUrl) { 8743 (*t)[idx] = (*t)[$-1]; 8744 (*t) = (*t)[0 .. $-1]; 8745 break; 8746 } 8747 } 8748 8749 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8750 if(receivedFd != -1) { 8751 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8752 8753 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8754 8755 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8756 8757 auto url = got.url.idup; 8758 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8759 8760 // FIXME: catch up on past messages here 8761 } else { 8762 auto data = op.usedBuffer; 8763 auto event = cast(SendableEvent*) data.ptr; 8764 8765 if(event.magic == 0xdeadbeef) { 8766 handleInputEvent(event); 8767 8768 if(event.url in pipes) 8769 foreach(pipe; pipes[event.url]) { 8770 event.url = pipe; 8771 handleInputEvent(event); 8772 } 8773 } else { 8774 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8775 } 8776 } 8777 return false; 8778 } 8779 void handleLocalConnectionClose(IoOp* op) { 8780 fileClosed(op.fd); 8781 } 8782 void handleLocalConnectionComplete(IoOp* op) {} 8783 8784 void wait_timeout() { 8785 // just keeping alive 8786 foreach(url, connections; eventConnectionsByUrl) 8787 foreach(connection; connections) 8788 if(connection.needsChunking) 8789 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8790 else 8791 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8792 } 8793 8794 void fileClosed(int fd) { 8795 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8796 foreach(idx, conn; connections) { 8797 if(fd == conn.fd) { 8798 connections[idx] = connections[$-1]; 8799 connections = connections[0 .. $ - 1]; 8800 continue outer; 8801 } 8802 } 8803 } 8804 } 8805 8806 void epoll_fd(int fd) {} 8807 8808 8809 private: 8810 8811 8812 struct SendableEventConnection { 8813 ubyte responseChunked; 8814 8815 int urlLength; 8816 char[256] urlBuffer = 0; 8817 8818 int lastEventIdLength; 8819 char[32] lastEventIdBuffer = 0; 8820 8821 char[] url() return { 8822 return urlBuffer[0 .. urlLength]; 8823 } 8824 void url(in char[] u) { 8825 urlBuffer[0 .. u.length] = u[]; 8826 urlLength = cast(int) u.length; 8827 } 8828 char[] lastEventId() return { 8829 return lastEventIdBuffer[0 .. lastEventIdLength]; 8830 } 8831 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8832 in { 8833 assert(url.length < this.urlBuffer.length); 8834 assert(lastEventId.length < this.lastEventIdBuffer.length); 8835 } 8836 do { 8837 this.responseChunked = responseChunked ? 1 : 0; 8838 this.urlLength = cast(int) url.length; 8839 this.lastEventIdLength = cast(int) lastEventId.length; 8840 8841 this.urlBuffer[0 .. url.length] = url[]; 8842 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8843 } 8844 } 8845 8846 struct SendableEvent { 8847 int magic = 0xdeadbeef; 8848 int urlLength; 8849 char[256] urlBuffer = 0; 8850 int typeLength; 8851 char[32] typeBuffer = 0; 8852 int messageLength; 8853 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8854 int _lifetime; 8855 8856 char[] message() return { 8857 return messageBuffer[0 .. messageLength]; 8858 } 8859 char[] type() return { 8860 return typeBuffer[0 .. typeLength]; 8861 } 8862 char[] url() return { 8863 return urlBuffer[0 .. urlLength]; 8864 } 8865 void url(in char[] u) { 8866 urlBuffer[0 .. u.length] = u[]; 8867 urlLength = cast(int) u.length; 8868 } 8869 int lifetime() { 8870 return _lifetime; 8871 } 8872 8873 /// 8874 void populate(string url, string type, string message, int lifetime) 8875 in { 8876 assert(url.length < this.urlBuffer.length); 8877 assert(type.length < this.typeBuffer.length); 8878 assert(message.length < this.messageBuffer.length); 8879 } 8880 do { 8881 this.urlLength = cast(int) url.length; 8882 this.typeLength = cast(int) type.length; 8883 this.messageLength = cast(int) message.length; 8884 this._lifetime = lifetime; 8885 8886 this.urlBuffer[0 .. url.length] = url[]; 8887 this.typeBuffer[0 .. type.length] = type[]; 8888 this.messageBuffer[0 .. message.length] = message[]; 8889 } 8890 } 8891 8892 struct EventConnection { 8893 int fd; 8894 bool needsChunking; 8895 } 8896 8897 private EventConnection[][string] eventConnectionsByUrl; 8898 private string[][string] pipes; 8899 8900 private void handleInputEvent(scope SendableEvent* event) { 8901 static int eventId; 8902 8903 static struct StoredEvent { 8904 int id; 8905 string type; 8906 string message; 8907 int lifetimeRemaining; 8908 } 8909 8910 StoredEvent[][string] byUrl; 8911 8912 int thisId = ++eventId; 8913 8914 if(event.lifetime) 8915 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8916 8917 auto connectionsPtr = event.url in eventConnectionsByUrl; 8918 EventConnection[] connections; 8919 if(connectionsPtr is null) 8920 return; 8921 else 8922 connections = *connectionsPtr; 8923 8924 char[4096] buffer; 8925 char[] formattedMessage; 8926 8927 void append(const char[] a) { 8928 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8929 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8930 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8931 } 8932 8933 import std.algorithm.iteration; 8934 8935 if(connections.length) { 8936 append("id: "); 8937 append(to!string(thisId)); 8938 append("\n"); 8939 8940 append("event: "); 8941 append(event.type); 8942 append("\n"); 8943 8944 foreach(line; event.message.splitter("\n")) { 8945 append("data: "); 8946 append(line); 8947 append("\n"); 8948 } 8949 8950 append("\n"); 8951 } 8952 8953 // chunk it for HTTP! 8954 auto len = toHex(formattedMessage.length); 8955 buffer[4 .. 6] = "\r\n"[]; 8956 buffer[4 - len.length .. 4] = len[]; 8957 buffer[6 + formattedMessage.length] = '\r'; 8958 buffer[6 + formattedMessage.length + 1] = '\n'; 8959 8960 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8961 // done 8962 8963 // FIXME: send back requests when needed 8964 // FIXME: send a single ":\n" every 15 seconds to keep alive 8965 8966 foreach(connection; connections) { 8967 if(connection.needsChunking) { 8968 nonBlockingWrite(this, connection.fd, chunkedMessage); 8969 } else { 8970 nonBlockingWrite(this, connection.fd, formattedMessage); 8971 } 8972 } 8973 } 8974 } 8975 8976 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8977 version(Posix) { 8978 8979 import core.sys.posix.unistd; 8980 import core.sys.posix.fcntl; 8981 import core.sys.posix.sys.un; 8982 8983 import core.sys.posix.signal; 8984 signal(SIGPIPE, SIG_IGN); 8985 8986 static extern(C) void sigchldhandler(int) { 8987 int status; 8988 import w = core.sys.posix.sys.wait; 8989 w.wait(&status); 8990 } 8991 signal(SIGCHLD, &sigchldhandler); 8992 8993 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8994 if(sock == -1) 8995 throw new Exception("socket " ~ to!string(errno)); 8996 8997 scope(failure) 8998 close(sock); 8999 9000 cloexec(sock); 9001 9002 // add-on server processes are assumed to be local, and thus will 9003 // use unix domain sockets. Besides, I want to pass sockets to them, 9004 // so it basically must be local (except for the session server, but meh). 9005 sockaddr_un addr; 9006 addr.sun_family = AF_UNIX; 9007 version(linux) { 9008 // on linux, we will use the abstract namespace 9009 addr.sun_path[0] = 0; 9010 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 9011 } else { 9012 // but otherwise, just use a file cuz we must. 9013 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 9014 } 9015 9016 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 9017 throw new Exception("bind " ~ to!string(errno)); 9018 9019 if(listen(sock, 128) == -1) 9020 throw new Exception("listen " ~ to!string(errno)); 9021 9022 makeNonBlocking(sock); 9023 9024 version(linux) { 9025 import core.sys.linux.epoll; 9026 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 9027 if(epoll_fd == -1) 9028 throw new Exception("epoll_create1 " ~ to!string(errno)); 9029 scope(failure) 9030 close(epoll_fd); 9031 } else { 9032 import core.sys.posix.poll; 9033 } 9034 9035 version(linux) 9036 eis.epoll_fd = epoll_fd; 9037 9038 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 9039 scope(exit) 9040 freeIoOp(acceptOp); 9041 9042 version(linux) { 9043 epoll_event ev; 9044 ev.events = EPOLLIN | EPOLLET; 9045 ev.data.ptr = acceptOp; 9046 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 9047 throw new Exception("epoll_ctl " ~ to!string(errno)); 9048 9049 epoll_event[64] events; 9050 } else { 9051 pollfd[] pollfds; 9052 IoOp*[int] ioops; 9053 pollfds ~= pollfd(sock, POLLIN); 9054 ioops[sock] = acceptOp; 9055 } 9056 9057 import core.time : MonoTime, seconds; 9058 9059 MonoTime timeout = MonoTime.currTime + 15.seconds; 9060 9061 while(true) { 9062 9063 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 9064 9065 int timeout_milliseconds = 0; // -1; // infinite 9066 9067 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 9068 if(timeout_milliseconds < 0) 9069 timeout_milliseconds = 0; 9070 9071 //writeln("waiting for ", name); 9072 9073 version(linux) { 9074 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 9075 if(nfds == -1) { 9076 if(errno == EINTR) 9077 continue; 9078 throw new Exception("epoll_wait " ~ to!string(errno)); 9079 } 9080 } else { 9081 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 9082 size_t lastIdx = 0; 9083 } 9084 9085 if(nfds == 0) { 9086 eis.wait_timeout(); 9087 timeout += 15.seconds; 9088 } 9089 9090 foreach(idx; 0 .. nfds) { 9091 version(linux) { 9092 auto flags = events[idx].events; 9093 auto ioop = cast(IoOp*) events[idx].data.ptr; 9094 } else { 9095 IoOp* ioop; 9096 foreach(tidx, thing; pollfds[lastIdx .. $]) { 9097 if(thing.revents) { 9098 ioop = ioops[thing.fd]; 9099 lastIdx += tidx + 1; 9100 break; 9101 } 9102 } 9103 } 9104 9105 //writeln(flags, " ", ioop.fd); 9106 9107 void newConnection() { 9108 // on edge triggering, it is important that we get it all 9109 while(true) { 9110 auto size = cast(socklen_t) addr.sizeof; 9111 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 9112 if(ns == -1) { 9113 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9114 // all done, got it all 9115 break; 9116 } 9117 throw new Exception("accept " ~ to!string(errno)); 9118 } 9119 cloexec(ns); 9120 9121 makeNonBlocking(ns); 9122 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 9123 niop.closeHandler = &eis.handleLocalConnectionClose; 9124 niop.completeHandler = &eis.handleLocalConnectionComplete; 9125 scope(failure) freeIoOp(niop); 9126 9127 version(linux) { 9128 epoll_event nev; 9129 nev.events = EPOLLIN | EPOLLET; 9130 nev.data.ptr = niop; 9131 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 9132 throw new Exception("epoll_ctl " ~ to!string(errno)); 9133 } else { 9134 bool found = false; 9135 foreach(ref pfd; pollfds) { 9136 if(pfd.fd < 0) { 9137 pfd.fd = ns; 9138 found = true; 9139 } 9140 } 9141 if(!found) 9142 pollfds ~= pollfd(ns, POLLIN); 9143 ioops[ns] = niop; 9144 } 9145 } 9146 } 9147 9148 bool newConnectionCondition() { 9149 version(linux) 9150 return ioop.fd == sock && (flags & EPOLLIN); 9151 else 9152 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 9153 } 9154 9155 if(newConnectionCondition()) { 9156 newConnection(); 9157 } else if(ioop.operation == IoOp.ReadSocketHandle) { 9158 while(true) { 9159 int in_fd; 9160 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 9161 if(got == -1) { 9162 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9163 // all done, got it all 9164 if(ioop.completeHandler) 9165 ioop.completeHandler(ioop); 9166 break; 9167 } 9168 throw new Exception("recv " ~ to!string(errno)); 9169 } 9170 9171 if(got == 0) { 9172 if(ioop.closeHandler) { 9173 ioop.closeHandler(ioop); 9174 version(linux) {} // nothing needed 9175 else { 9176 foreach(ref pfd; pollfds) { 9177 if(pfd.fd == ioop.fd) 9178 pfd.fd = -1; 9179 } 9180 } 9181 } 9182 close(ioop.fd); 9183 freeIoOp(ioop); 9184 break; 9185 } 9186 9187 ioop.bufferLengthUsed = cast(int) got; 9188 ioop.handler(ioop, in_fd); 9189 } 9190 } else if(ioop.operation == IoOp.Read) { 9191 while(true) { 9192 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 9193 if(got == -1) { 9194 if(errno == EAGAIN || errno == EWOULDBLOCK) { 9195 // all done, got it all 9196 if(ioop.completeHandler) 9197 ioop.completeHandler(ioop); 9198 break; 9199 } 9200 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 9201 } 9202 9203 if(got == 0) { 9204 if(ioop.closeHandler) 9205 ioop.closeHandler(ioop); 9206 close(ioop.fd); 9207 freeIoOp(ioop); 9208 break; 9209 } 9210 9211 ioop.bufferLengthUsed = cast(int) got; 9212 if(ioop.handler(ioop, ioop.fd)) { 9213 close(ioop.fd); 9214 freeIoOp(ioop); 9215 break; 9216 } 9217 } 9218 } 9219 9220 // EPOLLHUP? 9221 } 9222 } 9223 } else version(Windows) { 9224 9225 // set up a named pipe 9226 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 9227 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 9228 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 9229 9230 } else static assert(0); 9231 } 9232 9233 9234 version(with_sendfd) 9235 // copied from the web and ported from C 9236 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 9237 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 9238 msghdr msg; 9239 iovec[1] iov; 9240 9241 version(OSX) { 9242 //msg.msg_accrights = cast(cattr_t) &sendfd; 9243 //msg.msg_accrightslen = int.sizeof; 9244 } else version(Android) { 9245 } else { 9246 union ControlUnion { 9247 cmsghdr cm; 9248 char[CMSG_SPACE(int.sizeof)] control; 9249 } 9250 9251 ControlUnion control_un; 9252 cmsghdr* cmptr; 9253 9254 msg.msg_control = control_un.control.ptr; 9255 msg.msg_controllen = control_un.control.length; 9256 9257 cmptr = CMSG_FIRSTHDR(&msg); 9258 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 9259 cmptr.cmsg_level = SOL_SOCKET; 9260 cmptr.cmsg_type = SCM_RIGHTS; 9261 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 9262 } 9263 9264 msg.msg_name = null; 9265 msg.msg_namelen = 0; 9266 9267 iov[0].iov_base = ptr; 9268 iov[0].iov_len = nbytes; 9269 msg.msg_iov = iov.ptr; 9270 msg.msg_iovlen = 1; 9271 9272 return sendmsg(fd, &msg, 0); 9273 } 9274 9275 version(with_sendfd) 9276 // copied from the web and ported from C 9277 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9278 msghdr msg; 9279 iovec[1] iov; 9280 ssize_t n; 9281 int newfd; 9282 9283 version(OSX) { 9284 //msg.msg_accrights = cast(cattr_t) recvfd; 9285 //msg.msg_accrightslen = int.sizeof; 9286 } else version(Android) { 9287 } else { 9288 union ControlUnion { 9289 cmsghdr cm; 9290 char[CMSG_SPACE(int.sizeof)] control; 9291 } 9292 ControlUnion control_un; 9293 cmsghdr* cmptr; 9294 9295 msg.msg_control = control_un.control.ptr; 9296 msg.msg_controllen = control_un.control.length; 9297 } 9298 9299 msg.msg_name = null; 9300 msg.msg_namelen = 0; 9301 9302 iov[0].iov_base = ptr; 9303 iov[0].iov_len = nbytes; 9304 msg.msg_iov = iov.ptr; 9305 msg.msg_iovlen = 1; 9306 9307 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9308 return n; 9309 9310 version(OSX) { 9311 //if(msg.msg_accrightslen != int.sizeof) 9312 //*recvfd = -1; 9313 } else version(Android) { 9314 } else { 9315 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9316 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9317 if (cmptr.cmsg_level != SOL_SOCKET) 9318 throw new Exception("control level != SOL_SOCKET"); 9319 if (cmptr.cmsg_type != SCM_RIGHTS) 9320 throw new Exception("control type != SCM_RIGHTS"); 9321 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9322 } else 9323 *recvfd = -1; /* descriptor was not passed */ 9324 } 9325 9326 return n; 9327 } 9328 /* end read_fd */ 9329 9330 9331 /* 9332 Event source stuff 9333 9334 The api is: 9335 9336 sendEvent(string url, string type, string data, int timeout = 60*10); 9337 9338 attachEventListener(string url, int fd, lastId) 9339 9340 9341 It just sends to all attached listeners, and stores it until the timeout 9342 for replaying via lastEventId. 9343 */ 9344 9345 /* 9346 Session process stuff 9347 9348 it stores it all. the cgi object has a session object that can grab it 9349 9350 session may be done in the same process if possible, there is a version 9351 switch to choose if you want to override. 9352 */ 9353 9354 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9355 alias handler = dispatchHandler; 9356 string urlPrefix; 9357 bool rejectFurther; 9358 immutable(DispatcherDetails) details; 9359 } 9360 9361 private string urlify(string name) pure { 9362 return beautify(name, '-', true); 9363 } 9364 9365 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9366 if(name == "id") 9367 return allLowerCase ? name : "ID"; 9368 9369 char[160] buffer; 9370 int bufferIndex = 0; 9371 bool shouldCap = true; 9372 bool shouldSpace; 9373 bool lastWasCap; 9374 foreach(idx, char ch; name) { 9375 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9376 9377 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9378 if(lastWasCap) { 9379 // two caps in a row, don't change. Prolly acronym. 9380 } else { 9381 if(idx) 9382 shouldSpace = true; // new word, add space 9383 } 9384 9385 lastWasCap = true; 9386 } else { 9387 lastWasCap = false; 9388 } 9389 9390 if(shouldSpace) { 9391 buffer[bufferIndex++] = space; 9392 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9393 shouldSpace = false; 9394 } 9395 if(shouldCap) { 9396 if(ch >= 'a' && ch <= 'z') 9397 ch -= 32; 9398 shouldCap = false; 9399 } 9400 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9401 ch += 32; 9402 buffer[bufferIndex++] = ch; 9403 } 9404 return buffer[0 .. bufferIndex].idup; 9405 } 9406 9407 /* 9408 string urlFor(alias func)() { 9409 return __traits(identifier, func); 9410 } 9411 */ 9412 9413 /++ 9414 UDA: The name displayed to the user in auto-generated HTML. 9415 9416 Default is `beautify(identifier)`. 9417 +/ 9418 struct DisplayName { 9419 string name; 9420 } 9421 9422 /++ 9423 UDA: The name used in the URL or web parameter. 9424 9425 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9426 +/ 9427 struct UrlName { 9428 string name; 9429 } 9430 9431 /++ 9432 UDA: default format to respond for this method 9433 +/ 9434 struct DefaultFormat { string value; } 9435 9436 class MissingArgumentException : Exception { 9437 string functionName; 9438 string argumentName; 9439 string argumentType; 9440 9441 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9442 this.functionName = functionName; 9443 this.argumentName = argumentName; 9444 this.argumentType = argumentType; 9445 9446 super("Missing Argument: " ~ this.argumentName, file, line, next); 9447 } 9448 } 9449 9450 /++ 9451 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9452 9453 History: 9454 Added December 15, 2021 (dub v10.5) 9455 +/ 9456 class ResourceNotFoundException : Exception { 9457 string resourceType; 9458 string resourceId; 9459 9460 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9461 this.resourceType = resourceType; 9462 this.resourceId = resourceId; 9463 9464 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9465 } 9466 9467 } 9468 9469 /++ 9470 This can be attached to any constructor or function called from the cgi system. 9471 9472 If it is present, the function argument can NOT be set from web params, but instead 9473 is set to the return value of the given `func`. 9474 9475 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9476 the current request. Otherwise, it must take zero arguments. 9477 9478 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9479 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9480 the cgi object. 9481 9482 const arguments are also supported. 9483 +/ 9484 struct ifCalledFromWeb(alias func) {} 9485 9486 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9487 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9488 9489 // FIXME: any array of structs should also be settable or gettable from csv as well. 9490 9491 // FIXME: think more about checkboxes and bools. 9492 9493 import std.traits; 9494 9495 Parameters!method params; 9496 alias idents = ParameterIdentifierTuple!method; 9497 alias defaults = ParameterDefaults!method; 9498 9499 const(string)[] names; 9500 const(string)[] values; 9501 9502 // first, check for missing arguments and initialize to defaults if necessary 9503 9504 static if(is(typeof(method) P == __parameters)) 9505 foreach(idx, param; P) {{ 9506 // see: mustNotBeSetFromWebParams 9507 static if(is(param : Cgi)) { 9508 static assert(!is(param == immutable)); 9509 cast() params[idx] = cgi; 9510 } else static if(is(param == Session!D, D)) { 9511 static assert(!is(param == immutable)); 9512 cast() params[idx] = cgi.getSessionObject!D(); 9513 } else { 9514 bool populated; 9515 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9516 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9517 static if(is(typeof(func(cgi)))) 9518 params[idx] = func(cgi); 9519 else 9520 params[idx] = func(); 9521 9522 populated = true; 9523 } 9524 } 9525 9526 if(!populated) { 9527 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9528 params[idx] = param.getAutomaticallyForCgi(cgi); 9529 populated = true; 9530 } 9531 } 9532 9533 if(!populated) { 9534 auto ident = idents[idx]; 9535 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9536 if(ident !in cgi.get) { 9537 static if(is(defaults[idx] == void)) { 9538 static if(is(param == bool)) 9539 params[idx] = false; 9540 else 9541 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9542 } else 9543 params[idx] = defaults[idx]; 9544 } 9545 } else { 9546 if(ident !in cgi.post) { 9547 static if(is(defaults[idx] == void)) { 9548 static if(is(param == bool)) 9549 params[idx] = false; 9550 else 9551 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9552 } else 9553 params[idx] = defaults[idx]; 9554 } 9555 } 9556 } 9557 } 9558 }} 9559 9560 // second, parse the arguments in order to build up arrays, etc. 9561 9562 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9563 static if(is(T == struct)) { 9564 if(name == paramName) { 9565 *what = T.init; 9566 return true; 9567 } else { 9568 // could be a child. gonna allow either obj.field OR obj[field] 9569 9570 string afterName; 9571 9572 if(name[paramName.length] == '[') { 9573 int count = 1; 9574 auto idx = paramName.length + 1; 9575 while(idx < name.length && count > 0) { 9576 if(name[idx] == '[') 9577 count++; 9578 else if(name[idx] == ']') { 9579 count--; 9580 if(count == 0) break; 9581 } 9582 idx++; 9583 } 9584 9585 if(idx == name.length) 9586 return false; // malformed 9587 9588 auto insideBrackets = name[paramName.length + 1 .. idx]; 9589 afterName = name[idx + 1 .. $]; 9590 9591 name = name[0 .. paramName.length]; 9592 9593 paramName = insideBrackets; 9594 9595 } else if(name[paramName.length] == '.') { 9596 paramName = name[paramName.length + 1 .. $]; 9597 name = paramName; 9598 int p = 0; 9599 foreach(ch; paramName) { 9600 if(ch == '.' || ch == '[') 9601 break; 9602 p++; 9603 } 9604 9605 afterName = paramName[p .. $]; 9606 paramName = paramName[0 .. p]; 9607 } else { 9608 return false; 9609 } 9610 9611 if(paramName.length) 9612 // set the child member 9613 switch(paramName) { 9614 foreach(idx, memberName; __traits(allMembers, T)) 9615 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9616 // data member! 9617 case memberName: 9618 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9619 } 9620 default: 9621 // ok, not a member 9622 } 9623 } 9624 9625 return false; 9626 } else static if(is(T == enum)) { 9627 *what = to!T(value); 9628 return true; 9629 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9630 *what = to!T(value); 9631 return true; 9632 } else static if(is(T == bool)) { 9633 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9634 return true; 9635 } else static if(is(T == K[], K)) { 9636 K tmp; 9637 if(name == paramName) { 9638 // direct - set and append 9639 if(setVariable(name, paramName, &tmp, value)) { 9640 (*what) ~= tmp; 9641 return true; 9642 } else { 9643 return false; 9644 } 9645 } else { 9646 // child, append to last element 9647 // FIXME: what about range violations??? 9648 auto ptr = &(*what)[(*what).length - 1]; 9649 return setVariable(name, paramName, ptr, value); 9650 9651 } 9652 } else static if(is(T == V[K], K, V)) { 9653 // assoc array, name[key] is valid 9654 if(name == paramName) { 9655 // no action necessary 9656 return true; 9657 } else if(name[paramName.length] == '[') { 9658 int count = 1; 9659 auto idx = paramName.length + 1; 9660 while(idx < name.length && count > 0) { 9661 if(name[idx] == '[') 9662 count++; 9663 else if(name[idx] == ']') { 9664 count--; 9665 if(count == 0) break; 9666 } 9667 idx++; 9668 } 9669 if(idx == name.length) 9670 return false; // malformed 9671 9672 auto insideBrackets = name[paramName.length + 1 .. idx]; 9673 auto afterName = name[idx + 1 .. $]; 9674 9675 auto k = to!K(insideBrackets); 9676 V v; 9677 if(auto ptr = k in *what) 9678 v = *ptr; 9679 9680 name = name[0 .. paramName.length]; 9681 //writeln(name, afterName, " ", paramName); 9682 9683 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9684 if(ret) { 9685 (*what)[k] = v; 9686 return true; 9687 } 9688 } 9689 9690 return false; 9691 } else { 9692 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9693 } 9694 9695 //return false; 9696 } 9697 9698 void setArgument(string name, string value) { 9699 int p; 9700 foreach(ch; name) { 9701 if(ch == '.' || ch == '[') 9702 break; 9703 p++; 9704 } 9705 9706 auto paramName = name[0 .. p]; 9707 9708 sw: switch(paramName) { 9709 static if(is(typeof(method) P == __parameters)) 9710 foreach(idx, param; P) { 9711 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9712 // cannot be set from the outside 9713 } else { 9714 case idents[idx]: 9715 static if(is(param == Cgi.UploadedFile)) { 9716 params[idx] = cgi.files[name]; 9717 } else static if(is(param : const Cgi.UploadedFile[])) { 9718 (cast() params[idx]) = cgi.filesArray[name]; 9719 } else { 9720 setVariable(name, paramName, ¶ms[idx], value); 9721 } 9722 break sw; 9723 } 9724 } 9725 default: 9726 // ignore; not relevant argument 9727 } 9728 } 9729 9730 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9731 names = cgi.allGetNamesInOrder; 9732 values = cgi.allGetValuesInOrder; 9733 } else { 9734 names = cgi.allPostNamesInOrder; 9735 values = cgi.allPostValuesInOrder; 9736 } 9737 9738 foreach(idx, name; names) { 9739 setArgument(name, values[idx]); 9740 } 9741 9742 static if(is(ReturnType!method == void)) { 9743 typeof(null) ret; 9744 dg(params); 9745 } else { 9746 auto ret = dg(params); 9747 } 9748 9749 // FIXME: format return values 9750 // options are: json, html, csv. 9751 // also may need to wrap in envelope format: none, html, or json. 9752 return ret; 9753 } 9754 9755 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9756 static if(is(T : const(Cgi))) { 9757 return true; 9758 } else static if(is(T : const(Session!D), D)) { 9759 return true; 9760 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9761 return true; 9762 } else { 9763 foreach(uda; attrs) 9764 static if(is(uda == ifCalledFromWeb!func, alias func)) 9765 return true; 9766 return false; 9767 } 9768 } 9769 9770 private bool hasIfCalledFromWeb(attrs...)() { 9771 foreach(uda; attrs) 9772 static if(is(uda == ifCalledFromWeb!func, alias func)) 9773 return true; 9774 return false; 9775 } 9776 9777 /++ 9778 Implies POST path for the thing itself, then GET will get the automatic form. 9779 9780 The given customizer, if present, will be called as a filter on the Form object. 9781 9782 History: 9783 Added December 27, 2020 9784 +/ 9785 template AutomaticForm(alias customizer) { } 9786 9787 /++ 9788 This is meant to be returned by a function that takes a form POST submission. You 9789 want to set the url of the new resource it created, which is set as the http 9790 Location header for a "201 Created" result, and you can also set a separate 9791 destination for browser users, which it sets via a "Refresh" header. 9792 9793 The `resourceRepresentation` should generally be the thing you just created, and 9794 it will be the body of the http response when formatted through the presenter. 9795 The exact thing is up to you - it could just return an id, or the whole object, or 9796 perhaps a partial object. 9797 9798 Examples: 9799 --- 9800 class Test : WebObject { 9801 @(Cgi.RequestMethod.POST) 9802 CreatedResource!int makeThing(string value) { 9803 return CreatedResource!int(value.to!int, "/resources/id"); 9804 } 9805 } 9806 --- 9807 9808 History: 9809 Added December 18, 2021 9810 +/ 9811 struct CreatedResource(T) { 9812 static if(!is(T == void)) 9813 T resourceRepresentation; 9814 string resourceUrl; 9815 string refreshUrl; 9816 } 9817 9818 /+ 9819 /++ 9820 This can be attached as a UDA to a handler to add a http Refresh header on a 9821 successful run. (It will not be attached if the function throws an exception.) 9822 This will refresh the browser the given number of seconds after the page loads, 9823 to the url returned by `urlFunc`, which can be either a static function or a 9824 member method of the current handler object. 9825 9826 You might use this for a POST handler that is normally used from ajax, but you 9827 want it to degrade gracefully to a temporarily flashed message before reloading 9828 the main page. 9829 9830 History: 9831 Added December 18, 2021 9832 +/ 9833 struct Refresh(alias urlFunc) { 9834 int waitInSeconds; 9835 9836 string url() { 9837 static if(__traits(isStaticFunction, urlFunc)) 9838 return urlFunc(); 9839 else static if(is(urlFunc : string)) 9840 return urlFunc; 9841 } 9842 } 9843 +/ 9844 9845 /+ 9846 /++ 9847 Sets a filter to be run before 9848 9849 A before function can do validations of params and log and stop the function from running. 9850 +/ 9851 template Before(alias b) {} 9852 template After(alias b) {} 9853 +/ 9854 9855 /+ 9856 Argument conversions: for the most part, it is to!Thing(string). 9857 9858 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9859 they are passed 9860 9861 arr=foo&arr=bar <-- notice the same name. 9862 9863 Structs are first declared with an empty thing, then have their members set individually, 9864 with dot notation. The members are not required, just the initial declaration. 9865 9866 struct Foo { 9867 int a; 9868 string b; 9869 } 9870 void test(Foo foo){} 9871 9872 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9873 9874 Arrays of structs use this declaration. 9875 9876 void test(Foo[] foo) {} 9877 9878 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9879 9880 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9881 declaration is ignored. 9882 9883 Mind that order matters! The declaration MUST come first in the string. 9884 9885 Arrays of struct members follow this rule recursively. 9886 9887 struct Foo { 9888 int[] a; 9889 } 9890 9891 foo&foo.a=1&foo.a=2&foo&foo.a=1 9892 9893 9894 Associative arrays are formatted with brackets, after a declaration, like structs: 9895 9896 foo&foo[key]=value&foo[other_key]=value 9897 9898 9899 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9900 do not support the strict ordering requirements to work with these struct protocols. 9901 9902 FIXME: also perhaps accept application/json to better work with outside trash. 9903 9904 9905 Return values are also auto-formatted according to user-requested type: 9906 for json, it loops over and converts. 9907 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9908 +/ 9909 9910 /++ 9911 A web presenter is responsible for rendering things to HTML to be usable 9912 in a web browser. 9913 9914 They are passed as template arguments to the base classes of [WebObject] 9915 9916 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9917 and override it. Use forwarding and specialization to customize it. 9918 9919 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9920 class templates and your overridden templates work with each other. 9921 9922 --- 9923 class MyPresenter : WebPresenter!(MyPresenter) { 9924 @Override 9925 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9926 // present the CustomType 9927 } 9928 @Override 9929 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9930 // handle everything else via the super class, which will call 9931 // back to your class when appropriate 9932 super.presentSuccessfulReturnAsHtml(cgi, ret); 9933 } 9934 } 9935 --- 9936 9937 The meta argument in there can be overridden by your own facility. 9938 9939 +/ 9940 class WebPresenter(CRTP) { 9941 9942 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9943 /// If you override a plain method, use `override`. If a template, use `@Override`. 9944 enum Override; 9945 9946 string script() { 9947 return ` 9948 `; 9949 } 9950 9951 string style() { 9952 return ` 9953 :root { 9954 --mild-border: #ccc; 9955 --middle-border: #999; 9956 --accent-color: #f2f2f2; 9957 --sidebar-color: #fefefe; 9958 } 9959 ` ~ genericFormStyling() ~ genericSiteStyling(); 9960 } 9961 9962 string genericFormStyling() { 9963 return 9964 q"css 9965 table.automatic-data-display { 9966 border-collapse: collapse; 9967 border: solid 1px var(--mild-border); 9968 } 9969 9970 table.automatic-data-display td { 9971 vertical-align: top; 9972 border: solid 1px var(--mild-border); 9973 padding: 2px 4px; 9974 } 9975 9976 table.automatic-data-display th { 9977 border: solid 1px var(--mild-border); 9978 border-bottom: solid 1px var(--middle-border); 9979 padding: 2px 4px; 9980 } 9981 9982 ol.automatic-data-display { 9983 margin: 0px; 9984 list-style-position: inside; 9985 padding: 0px; 9986 } 9987 9988 dl.automatic-data-display { 9989 9990 } 9991 9992 .automatic-form { 9993 max-width: 600px; 9994 } 9995 9996 .form-field { 9997 margin: 0.5em; 9998 padding-left: 0.5em; 9999 } 10000 10001 .label-text { 10002 display: block; 10003 font-weight: bold; 10004 margin-left: -0.5em; 10005 } 10006 10007 .submit-button-holder { 10008 padding-left: 2em; 10009 } 10010 10011 .add-array-button { 10012 10013 } 10014 css"; 10015 } 10016 10017 string genericSiteStyling() { 10018 return 10019 q"css 10020 * { box-sizing: border-box; } 10021 html, body { margin: 0px; } 10022 body { 10023 font-family: sans-serif; 10024 } 10025 header { 10026 background: var(--accent-color); 10027 height: 64px; 10028 } 10029 footer { 10030 background: var(--accent-color); 10031 height: 64px; 10032 } 10033 #site-container { 10034 display: flex; 10035 flex-wrap: wrap; 10036 } 10037 main { 10038 flex: 1 1 auto; 10039 order: 2; 10040 min-height: calc(100vh - 64px - 64px); 10041 min-width: 80ch; 10042 padding: 4px; 10043 padding-left: 1em; 10044 } 10045 #sidebar { 10046 flex: 0 0 16em; 10047 order: 1; 10048 background: var(--sidebar-color); 10049 } 10050 css"; 10051 } 10052 10053 import arsd.dom; 10054 Element htmlContainer() { 10055 auto document = new Document(q"html 10056 <!DOCTYPE html> 10057 <html class="no-script"> 10058 <head> 10059 <script>document.documentElement.classList.remove("no-script");</script> 10060 <style>.no-script requires-script { display: none; }</style> 10061 <title>D Application</title> 10062 <meta name="viewport" content="initial-scale=1, width=device-width" /> 10063 <link rel="stylesheet" href="style.css" /> 10064 </head> 10065 <body> 10066 <header></header> 10067 <div id="site-container"> 10068 <main></main> 10069 <div id="sidebar"></div> 10070 </div> 10071 <footer></footer> 10072 <script src="script.js"></script> 10073 </body> 10074 </html> 10075 html", true, true); 10076 10077 return document.requireSelector("main"); 10078 } 10079 10080 /// Renders a response as an HTTP error with associated html body 10081 void renderBasicError(Cgi cgi, int httpErrorCode) { 10082 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 10083 auto c = htmlContainer(); 10084 c.innerText = getHttpCodeText(httpErrorCode); 10085 cgi.setResponseContentType("text/html; charset=utf-8"); 10086 cgi.write(c.parentDocument.toString(), true); 10087 } 10088 10089 template methodMeta(alias method) { 10090 enum methodMeta = null; 10091 } 10092 10093 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 10094 switch(format) { 10095 case "html": 10096 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 10097 break; 10098 case "json": 10099 import arsd.jsvar; 10100 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 10101 var json; 10102 foreach(index, type; Types) { 10103 if(ret.contains == index) 10104 json = ret.payload[index]; 10105 } 10106 } else { 10107 var json = ret; 10108 } 10109 var envelope = json; // var.emptyObject; 10110 /* 10111 envelope.success = true; 10112 envelope.result = json; 10113 envelope.error = null; 10114 */ 10115 cgi.setResponseContentType("application/json"); 10116 cgi.write(envelope.toJson(), true); 10117 break; 10118 default: 10119 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 10120 } 10121 } 10122 10123 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 10124 /// in the default presenter - allowing the function to have full low-level control over the 10125 /// response. 10126 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 10127 // nothing intentionally! 10128 } 10129 10130 /// Redirections are forwarded to [Cgi.setResponseLocation] 10131 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 10132 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 10133 } 10134 10135 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 10136 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 10137 cgi.setResponseStatus(getHttpCodeText(201)); 10138 if(ret.resourceUrl.length) 10139 cgi.header("Location: " ~ ret.resourceUrl); 10140 if(ret.refreshUrl.length) 10141 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 10142 static if(!is(R == void)) 10143 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 10144 } 10145 10146 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 10147 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 10148 bool outputted = false; 10149 foreach(index, type; Types) { 10150 if(ret.contains == index) { 10151 assert(!outputted); 10152 outputted = true; 10153 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 10154 } 10155 } 10156 if(!outputted) 10157 assert(0); 10158 } 10159 10160 /++ 10161 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 10162 +/ 10163 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 10164 cgi.setCache(true); // not necessarily true but meh 10165 if(auto fn = ret.filename()) { 10166 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 10167 } 10168 cgi.setResponseContentType(ret.contentType); 10169 cgi.write(ret.getData(), true); 10170 } 10171 10172 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 10173 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 10174 auto container = this.htmlContainer(); 10175 container.appendChild(formatReturnValueAsHtml(ret)); 10176 cgi.write(container.parentDocument.toString(), true); 10177 } 10178 10179 /++ 10180 10181 History: 10182 Added January 23, 2023 (dub v11.0) 10183 +/ 10184 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 10185 switch(format) { 10186 case "html": 10187 presentExceptionAsHtml(cgi, t, meta); 10188 break; 10189 case "json": 10190 presentExceptionAsJsonImpl(cgi, t); 10191 break; 10192 default: 10193 } 10194 } 10195 10196 private void presentExceptionAsJsonImpl()(Cgi cgi, Throwable t) { 10197 cgi.setResponseStatus("500 Internal Server Error"); 10198 cgi.setResponseContentType("application/json"); 10199 import arsd.jsvar; 10200 var v = var.emptyObject; 10201 v.type = typeid(t).toString; 10202 v.msg = t.msg; 10203 v.fullString = t.toString(); 10204 cgi.write(v.toJson(), true); 10205 } 10206 10207 10208 /++ 10209 If you override this, you will need to cast the exception type `t` dynamically, 10210 but can then use the template arguments here to refer back to the function. 10211 10212 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 10213 method on the live object. You could, in theory, change arguments and retry, but I 10214 provide that information mostly with the expectation that you will use them to make 10215 useful forms or richer error messages for the user. 10216 10217 History: 10218 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 10219 I removed this in favor of a `Meta` param. 10220 10221 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 10222 10223 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 10224 10225 If you used the func for something, move that something into your `methodMeta` template. 10226 10227 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 10228 enabling an easier implementation of [presentExceptionalReturn]. 10229 +/ 10230 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 10231 Form af; 10232 /+ 10233 foreach(attr; __traits(getAttributes, func)) { 10234 static if(__traits(isSame, attr, AutomaticForm)) { 10235 af = createAutomaticFormForFunction!(func)(dg); 10236 } 10237 } 10238 +/ 10239 presentExceptionAsHtmlImpl(cgi, t, af); 10240 } 10241 10242 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 10243 if(auto e = cast(ResourceNotFoundException) t) { 10244 auto container = this.htmlContainer(); 10245 10246 container.addChild("p", e.msg); 10247 10248 if(!cgi.outputtedResponseData) 10249 cgi.setResponseStatus("404 Not Found"); 10250 cgi.write(container.parentDocument.toString(), true); 10251 } else if(auto mae = cast(MissingArgumentException) t) { 10252 if(automaticForm is null) 10253 goto generic; 10254 auto container = this.htmlContainer(); 10255 if(cgi.requestMethod == Cgi.RequestMethod.POST) 10256 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 10257 container.appendChild(automaticForm); 10258 10259 cgi.write(container.parentDocument.toString(), true); 10260 } else { 10261 generic: 10262 auto container = this.htmlContainer(); 10263 10264 // import std.stdio; writeln(t.toString()); 10265 10266 container.appendChild(exceptionToElement(t)); 10267 10268 container.addChild("h4", "GET"); 10269 foreach(k, v; cgi.get) { 10270 auto deets = container.addChild("details"); 10271 deets.addChild("summary", k); 10272 deets.addChild("div", v); 10273 } 10274 10275 container.addChild("h4", "POST"); 10276 foreach(k, v; cgi.post) { 10277 auto deets = container.addChild("details"); 10278 deets.addChild("summary", k); 10279 deets.addChild("div", v); 10280 } 10281 10282 10283 if(!cgi.outputtedResponseData) 10284 cgi.setResponseStatus("500 Internal Server Error"); 10285 cgi.write(container.parentDocument.toString(), true); 10286 } 10287 } 10288 10289 Element exceptionToElement(Throwable t) { 10290 auto div = Element.make("div"); 10291 div.addClass("exception-display"); 10292 10293 div.addChild("p", t.msg); 10294 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10295 10296 auto pre = div.addChild("pre"); 10297 string s; 10298 s = t.toString(); 10299 Element currentBox; 10300 bool on = false; 10301 foreach(line; s.splitLines) { 10302 if(!on && line.startsWith("-----")) 10303 on = true; 10304 if(!on) continue; 10305 if(line.indexOf("arsd/") != -1) { 10306 if(currentBox is null) { 10307 currentBox = pre.addChild("details"); 10308 currentBox.addChild("summary", "Framework code"); 10309 } 10310 currentBox.addChild("span", line ~ "\n"); 10311 } else { 10312 pre.addChild("span", line ~ "\n"); 10313 currentBox = null; 10314 } 10315 } 10316 10317 return div; 10318 } 10319 10320 /++ 10321 Returns an element for a particular type 10322 +/ 10323 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10324 import std.traits; 10325 10326 auto div = Element.make("div"); 10327 div.addClass("form-field"); 10328 10329 static if(is(T : const Cgi.UploadedFile)) { 10330 Element lbl; 10331 if(displayName !is null) { 10332 lbl = div.addChild("label"); 10333 lbl.addChild("span", displayName, "label-text"); 10334 lbl.appendText(" "); 10335 } else { 10336 lbl = div; 10337 } 10338 auto i = lbl.addChild("input", name); 10339 i.attrs.name = name; 10340 i.attrs.type = "file"; 10341 i.attrs.multiple = "multiple"; 10342 } else static if(is(T == Cgi.UploadedFile)) { 10343 Element lbl; 10344 if(displayName !is null) { 10345 lbl = div.addChild("label"); 10346 lbl.addChild("span", displayName, "label-text"); 10347 lbl.appendText(" "); 10348 } else { 10349 lbl = div; 10350 } 10351 auto i = lbl.addChild("input", name); 10352 i.attrs.name = name; 10353 i.attrs.type = "file"; 10354 } else static if(is(T == enum)) { 10355 Element lbl; 10356 if(displayName !is null) { 10357 lbl = div.addChild("label"); 10358 lbl.addChild("span", displayName, "label-text"); 10359 lbl.appendText(" "); 10360 } else { 10361 lbl = div; 10362 } 10363 auto i = lbl.addChild("select", name); 10364 i.attrs.name = name; 10365 10366 foreach(memberName; __traits(allMembers, T)) 10367 i.addChild("option", memberName); 10368 10369 } else static if(is(T == struct)) { 10370 if(displayName !is null) 10371 div.addChild("span", displayName, "label-text"); 10372 auto fieldset = div.addChild("fieldset"); 10373 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10374 fieldset.addChild("input", name); 10375 foreach(idx, memberName; __traits(allMembers, T)) 10376 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10377 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10378 } 10379 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10380 Element lbl; 10381 if(displayName !is null) { 10382 lbl = div.addChild("label"); 10383 lbl.addChild("span", displayName, "label-text"); 10384 lbl.appendText(" "); 10385 } else { 10386 lbl = div; 10387 } 10388 Element i; 10389 if(udaSuggestion) { 10390 i = udaSuggestion(); 10391 lbl.appendChild(i); 10392 } else { 10393 i = lbl.addChild("input", name); 10394 } 10395 i.attrs.name = name; 10396 static if(isSomeString!T) 10397 i.attrs.type = "text"; 10398 else 10399 i.attrs.type = "number"; 10400 if(i.tagName == "textarea") 10401 i.textContent = to!string(T.init); 10402 else 10403 i.attrs.value = to!string(T.init); 10404 } else static if(is(T == bool)) { 10405 Element lbl; 10406 if(displayName !is null) { 10407 lbl = div.addChild("label"); 10408 lbl.addChild("span", displayName, "label-text"); 10409 lbl.appendText(" "); 10410 } else { 10411 lbl = div; 10412 } 10413 auto i = lbl.addChild("input", name); 10414 i.attrs.type = "checkbox"; 10415 i.attrs.value = "true"; 10416 i.attrs.name = name; 10417 } else static if(is(T == K[], K)) { 10418 auto templ = div.addChild("template"); 10419 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10420 if(displayName !is null) 10421 div.addChild("span", displayName, "label-text"); 10422 auto btn = div.addChild("button"); 10423 btn.addClass("add-array-button"); 10424 btn.attrs.type = "button"; 10425 btn.innerText = "Add"; 10426 btn.attrs.onclick = q{ 10427 var a = document.importNode(this.parentNode.firstChild.content, true); 10428 this.parentNode.insertBefore(a, this); 10429 }; 10430 } else static if(is(T == V[K], K, V)) { 10431 div.innerText = "assoc array not implemented for automatic form at this time"; 10432 } else { 10433 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10434 } 10435 10436 10437 return div; 10438 } 10439 10440 /// creates a form for gathering the function's arguments 10441 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10442 10443 auto form = cast(Form) Element.make("form"); 10444 10445 form.method = "POST"; // FIXME 10446 10447 form.addClass("automatic-form"); 10448 10449 string formDisplayName = beautify(__traits(identifier, method)); 10450 foreach(attr; __traits(getAttributes, method)) 10451 static if(is(typeof(attr) == DisplayName)) 10452 formDisplayName = attr.name; 10453 form.addChild("h3", formDisplayName); 10454 10455 import std.traits; 10456 10457 //Parameters!method params; 10458 //alias idents = ParameterIdentifierTuple!method; 10459 //alias defaults = ParameterDefaults!method; 10460 10461 static if(is(typeof(method) P == __parameters)) 10462 foreach(idx, _; P) {{ 10463 10464 alias param = P[idx .. idx + 1]; 10465 10466 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10467 string displayName = beautify(__traits(identifier, param)); 10468 Element function() element; 10469 foreach(attr; __traits(getAttributes, param)) { 10470 static if(is(typeof(attr) == DisplayName)) 10471 displayName = attr.name; 10472 else static if(is(typeof(attr) : typeof(element))) { 10473 element = attr; 10474 } 10475 } 10476 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10477 if(i.querySelector("input[type=file]") !is null) 10478 form.setAttribute("enctype", "multipart/form-data"); 10479 } 10480 }} 10481 10482 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10483 10484 return form; 10485 } 10486 10487 /// creates a form for gathering object members (for the REST object thing right now) 10488 Form createAutomaticFormForObject(T)(T obj) { 10489 auto form = cast(Form) Element.make("form"); 10490 10491 form.addClass("automatic-form"); 10492 10493 form.addChild("h3", beautify(__traits(identifier, T))); 10494 10495 import std.traits; 10496 10497 //Parameters!method params; 10498 //alias idents = ParameterIdentifierTuple!method; 10499 //alias defaults = ParameterDefaults!method; 10500 10501 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10502 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10503 string displayName = beautify(memberName); 10504 Element function() element; 10505 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10506 static if(is(typeof(attr) == DisplayName)) 10507 displayName = attr.name; 10508 else static if(is(typeof(attr) : typeof(element))) 10509 element = attr; 10510 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10511 10512 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10513 }}} 10514 10515 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10516 10517 return form; 10518 } 10519 10520 /// 10521 Element formatReturnValueAsHtml(T)(T t) { 10522 import std.traits; 10523 10524 static if(is(T == typeof(null))) { 10525 return Element.make("span"); 10526 } else static if(is(T : Element)) { 10527 return t; 10528 } else static if(is(T == MultipleResponses!Types, Types...)) { 10529 foreach(index, type; Types) { 10530 if(t.contains == index) 10531 return formatReturnValueAsHtml(t.payload[index]); 10532 } 10533 assert(0); 10534 } else static if(is(T == Paginated!E, E)) { 10535 auto e = Element.make("div").addClass("paginated-result"); 10536 e.appendChild(formatReturnValueAsHtml(t.items)); 10537 if(t.nextPageUrl.length) 10538 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10539 return e; 10540 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10541 return Element.make("span", to!string(t), "automatic-data-display"); 10542 } else static if(is(T == V[K], K, V)) { 10543 auto dl = Element.make("dl"); 10544 dl.addClass("automatic-data-display associative-array"); 10545 foreach(k, v; t) { 10546 dl.addChild("dt", to!string(k)); 10547 dl.addChild("dd", formatReturnValueAsHtml(v)); 10548 } 10549 return dl; 10550 } else static if(is(T == struct)) { 10551 auto dl = Element.make("dl"); 10552 dl.addClass("automatic-data-display struct"); 10553 10554 foreach(idx, memberName; __traits(allMembers, T)) 10555 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10556 dl.addChild("dt", beautify(memberName)); 10557 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10558 } 10559 10560 return dl; 10561 } else static if(is(T == bool)) { 10562 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10563 } else static if(is(T == E[], E)) { 10564 static if(is(E : RestObject!Proxy, Proxy)) { 10565 // treat RestObject similar to struct 10566 auto table = cast(Table) Element.make("table"); 10567 table.addClass("automatic-data-display"); 10568 string[] names; 10569 foreach(idx, memberName; __traits(derivedMembers, E)) 10570 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10571 names ~= beautify(memberName); 10572 } 10573 table.appendHeaderRow(names); 10574 10575 foreach(l; t) { 10576 auto tr = table.appendRow(); 10577 foreach(idx, memberName; __traits(derivedMembers, E)) 10578 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10579 static if(memberName == "id") { 10580 string val = to!string(__traits(getMember, l, memberName)); 10581 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10582 } else { 10583 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10584 } 10585 } 10586 } 10587 10588 return table; 10589 } else static if(is(E == struct)) { 10590 // an array of structs is kinda special in that I like 10591 // having those formatted as tables. 10592 auto table = cast(Table) Element.make("table"); 10593 table.addClass("automatic-data-display"); 10594 string[] names; 10595 foreach(idx, memberName; __traits(allMembers, E)) 10596 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10597 names ~= beautify(memberName); 10598 } 10599 table.appendHeaderRow(names); 10600 10601 foreach(l; t) { 10602 auto tr = table.appendRow(); 10603 foreach(idx, memberName; __traits(allMembers, E)) 10604 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10605 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10606 } 10607 } 10608 10609 return table; 10610 } else { 10611 // otherwise, I will just make a list. 10612 auto ol = Element.make("ol"); 10613 ol.addClass("automatic-data-display"); 10614 foreach(e; t) 10615 ol.addChild("li", formatReturnValueAsHtml(e)); 10616 return ol; 10617 } 10618 } else static if(is(T : Object)) { 10619 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10620 return Element.make("div", t.toHtml()); 10621 else 10622 return Element.make("div", t.toString()); 10623 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10624 10625 assert(0); 10626 } 10627 10628 } 10629 10630 /++ 10631 The base class for the [dispatcher] function and object support. 10632 +/ 10633 class WebObject { 10634 //protected Cgi cgi; 10635 10636 protected void initialize(Cgi cgi) { 10637 //this.cgi = cgi; 10638 } 10639 } 10640 10641 /++ 10642 Can return one of the given types, decided at runtime. The syntax 10643 is to declare all the possible types in the return value, then you 10644 can `return typeof(return)(...value...)` to construct it. 10645 10646 It has an auto-generated constructor for each value it can hold. 10647 10648 --- 10649 MultipleResponses!(Redirection, string) getData(int how) { 10650 if(how & 1) 10651 return typeof(return)(Redirection("http://dpldocs.info/")); 10652 else 10653 return typeof(return)("hi there!"); 10654 } 10655 --- 10656 10657 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10658 +/ 10659 struct MultipleResponses(T...) { 10660 private size_t contains; 10661 private union { 10662 private T payload; 10663 } 10664 10665 static foreach(index, type; T) 10666 public this(type t) { 10667 contains = index; 10668 payload[index] = t; 10669 } 10670 10671 /++ 10672 This is primarily for testing. It is your way of getting to the response. 10673 10674 Let's say you wanted to test that one holding a Redirection and a string actually 10675 holds a string, by name of "test": 10676 10677 --- 10678 auto valueToTest = your_test_function(); 10679 10680 valueToTest.visit( 10681 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10682 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10683 ); 10684 --- 10685 10686 History: 10687 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10688 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10689 +/ 10690 void visit(Handlers...)(Handlers handlers) { 10691 template findHandler(type, int count, HandlersToCheck...) { 10692 static if(HandlersToCheck.length == 0) 10693 enum findHandler = -1; 10694 else { 10695 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10696 enum findHandler = count; 10697 else 10698 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10699 } 10700 } 10701 foreach(index, type; T) { 10702 enum handlerIndex = findHandler!(type, 0, Handlers); 10703 static if(handlerIndex == -1) 10704 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10705 else { 10706 if(index == this.contains) 10707 handlers[handlerIndex](this.payload[index]); 10708 } 10709 } 10710 } 10711 10712 /+ 10713 auto toArsdJsvar()() { 10714 import arsd.jsvar; 10715 return var(null); 10716 } 10717 +/ 10718 } 10719 10720 // FIXME: implement this somewhere maybe 10721 struct RawResponse { 10722 int code; 10723 string[] headers; 10724 const(ubyte)[] responseBody; 10725 } 10726 10727 /++ 10728 You can return this from [WebObject] subclasses for redirections. 10729 10730 (though note the static types means that class must ALWAYS redirect if 10731 you return this directly. You might want to return [MultipleResponses] if it 10732 can be conditional) 10733 +/ 10734 struct Redirection { 10735 string to; /// The URL to redirect to. 10736 int code = 303; /// The HTTP code to return. 10737 } 10738 10739 /++ 10740 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10741 10742 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10743 the presenter in the dispatcher. 10744 10745 FIXME: explain this better 10746 10747 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10748 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10749 the runtime result of that is undefined. 10750 10751 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10752 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10753 10754 $(WARNING 10755 --- 10756 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10757 // even if you put different URL udas on it, the current code ignores them. 10758 void foo(int a) {} 10759 void foo(string a) {} 10760 --- 10761 ) 10762 10763 See_Also: [serveRestObject], [serveStaticFile] 10764 +/ 10765 auto serveApi(T)(string urlPrefix) { 10766 assert(urlPrefix[$ - 1] == '/'); 10767 return serveApiInternal!T(urlPrefix); 10768 } 10769 10770 private string nextPieceFromSlash(ref string remainingUrl) { 10771 if(remainingUrl.length == 0) 10772 return remainingUrl; 10773 int slash = 0; 10774 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10775 slash++; 10776 10777 // I am specifically passing `null` to differentiate it vs empty string 10778 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10779 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10780 // so if it is the last item, the dot can be used to load an alternative view 10781 // otherwise tho the dot is considered part of the identifier 10782 // FIXME 10783 10784 // again notice "" vs null here! 10785 if(slash == remainingUrl.length) 10786 remainingUrl = null; 10787 else 10788 remainingUrl = remainingUrl[slash + 1 .. $]; 10789 10790 return ident; 10791 } 10792 10793 /++ 10794 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10795 +/ 10796 enum AddTrailingSlash; 10797 /// ditto 10798 enum RemoveTrailingSlash; 10799 10800 private auto serveApiInternal(T)(string urlPrefix) { 10801 10802 import arsd.dom; 10803 import arsd.jsvar; 10804 10805 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10806 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10807 10808 try { 10809 // see duplicated code below by searching subresource_ctor 10810 // also see mustNotBeSetFromWebParams 10811 10812 static if(is(typeof(T.__ctor) P == __parameters)) { 10813 P params; 10814 10815 foreach(pidx, param; P) { 10816 static if(is(param : Cgi)) { 10817 static assert(!is(param == immutable)); 10818 cast() params[pidx] = cgi; 10819 } else static if(is(param == Session!D, D)) { 10820 static assert(!is(param == immutable)); 10821 cast() params[pidx] = cgi.getSessionObject!D(); 10822 10823 } else { 10824 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10825 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10826 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10827 static if(is(typeof(func(cgi)))) 10828 params[pidx] = func(cgi); 10829 else 10830 params[pidx] = func(); 10831 } 10832 } 10833 } else { 10834 10835 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10836 params[pidx] = param.getAutomaticallyForCgi(cgi); 10837 } else static if(is(param == string)) { 10838 auto ident = nextPieceFromSlash(remainingUrl); 10839 params[pidx] = ident; 10840 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10841 } 10842 } 10843 } 10844 10845 auto obj = new T(params); 10846 } else { 10847 auto obj = new T(); 10848 } 10849 10850 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10851 } catch(Throwable t) { 10852 switch(cgi.request("format", "html")) { 10853 case "html": 10854 static void dummy() {} 10855 presenter.presentExceptionAsHtml(cgi, t, null); 10856 return true; 10857 case "json": 10858 var envelope = var.emptyObject; 10859 envelope.success = false; 10860 envelope.result = null; 10861 envelope.error = t.toString(); 10862 cgi.setResponseContentType("application/json"); 10863 cgi.write(envelope.toJson(), true); 10864 return true; 10865 default: 10866 throw t; 10867 // return true; 10868 } 10869 // return true; 10870 } 10871 10872 assert(0); 10873 } 10874 10875 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10876 10877 obj.initialize(cgi); 10878 10879 /+ 10880 Overload rules: 10881 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10882 statically. 10883 10884 Moreover, some args vs no args can be overloaded dynamically. 10885 +/ 10886 10887 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10888 /+ 10889 auto orig = remainingUrl; 10890 assert(0, 10891 (orig is null ? "__null" : orig) 10892 ~ " .. " ~ 10893 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10894 +/ 10895 10896 if(methodNameFromUrl is null) 10897 methodNameFromUrl = "__null"; 10898 10899 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10900 10901 if(remainingUrl.length) 10902 hack ~= "/"; 10903 10904 switch(hack) { 10905 foreach(methodName; __traits(derivedMembers, T)) 10906 static if(methodName != "__ctor") 10907 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10908 static if(is(typeof(overload) P == __parameters)) 10909 static if(is(typeof(overload) R == return)) 10910 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10911 { 10912 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10913 case urlNameForMethod: 10914 10915 static if(is(R : WebObject)) { 10916 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10917 10918 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10919 10920 // subresource_ctor 10921 // also see mustNotBeSetFromWebParams 10922 10923 P params; 10924 10925 string ident; 10926 10927 foreach(pidx, param; P) { 10928 static if(is(param : Cgi)) { 10929 static assert(!is(param == immutable)); 10930 cast() params[pidx] = cgi; 10931 } else static if(is(param == typeof(presenter))) { 10932 cast() param[pidx] = presenter; 10933 } else static if(is(param == Session!D, D)) { 10934 static assert(!is(param == immutable)); 10935 cast() params[pidx] = cgi.getSessionObject!D(); 10936 } else { 10937 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10938 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10939 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10940 static if(is(typeof(func(cgi)))) 10941 params[pidx] = func(cgi); 10942 else 10943 params[pidx] = func(); 10944 } 10945 } 10946 } else { 10947 10948 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10949 params[pidx] = param.getAutomaticallyForCgi(cgi); 10950 } else static if(is(param == string)) { 10951 ident = nextPieceFromSlash(remainingUrl); 10952 if(ident is null) { 10953 // trailing slash mandated on subresources 10954 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10955 return true; 10956 } else { 10957 params[pidx] = ident; 10958 } 10959 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10960 } 10961 } 10962 } 10963 10964 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10965 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10966 } else { 10967 // 404 it if any url left - not a subresource means we don't get to play with that! 10968 if(remainingUrl.length) 10969 return false; 10970 10971 bool automaticForm; 10972 10973 foreach(attr; __traits(getAttributes, overload)) 10974 static if(is(attr == AddTrailingSlash)) { 10975 if(remainingUrl is null) { 10976 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10977 return true; 10978 } 10979 } else static if(is(attr == RemoveTrailingSlash)) { 10980 if(remainingUrl !is null) { 10981 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10982 return true; 10983 } 10984 10985 } else static if(__traits(isSame, AutomaticForm, attr)) { 10986 automaticForm = true; 10987 } 10988 10989 /+ 10990 int zeroArgOverload = -1; 10991 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10992 bool calledWithZeroArgs = true; 10993 foreach(k, v; cgi.get) 10994 if(k != "format") { 10995 calledWithZeroArgs = false; 10996 break; 10997 } 10998 foreach(k, v; cgi.post) 10999 if(k != "format") { 11000 calledWithZeroArgs = false; 11001 break; 11002 } 11003 11004 // first, we need to go through and see if there is an empty one, since that 11005 // changes inside. But otherwise, all the stuff I care about can be done via 11006 // simple looping (other improper overloads might be flagged for runtime semantic check) 11007 // 11008 // an argument of type Cgi is ignored for these purposes 11009 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 11010 static if(is(typeof(overload) P == __parameters)) 11011 static if(P.length == 0) 11012 zeroArgOverload = cast(int) idx; 11013 else static if(P.length == 1 && is(P[0] : Cgi)) 11014 zeroArgOverload = cast(int) idx; 11015 }} 11016 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 11017 bool overloadHasBeenCalled = false; 11018 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 11019 bool callFunction = true; 11020 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 11021 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 11022 callFunction = false; 11023 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 11024 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 11025 callFunction = false; 11026 11027 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 11028 11029 bool hadAnyMethodRestrictions = false; 11030 bool foundAcceptableMethod = false; 11031 foreach(attr; __traits(getAttributes, overload)) { 11032 static if(is(typeof(attr) == Cgi.RequestMethod)) { 11033 hadAnyMethodRestrictions = true; 11034 if(attr == cgi.requestMethod) 11035 foundAcceptableMethod = true; 11036 } 11037 } 11038 11039 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 11040 callFunction = false; 11041 11042 /+ 11043 The overloads we really want to allow are the sane ones 11044 from the web perspective. Which is likely on HTTP verbs, 11045 for the most part, but might also be potentially based on 11046 some args vs zero args, or on argument names. Can't really 11047 do argument types very reliable through the web though; those 11048 should probably be different URLs. 11049 11050 Even names I feel is better done inside the function, so I'm not 11051 going to support that here. But the HTTP verbs and zero vs some 11052 args makes sense - it lets you define custom forms pretty easily. 11053 11054 Moreover, I'm of the opinion that empty overload really only makes 11055 sense on GET for this case. On a POST, it is just a missing argument 11056 exception and that should be handled by the presenter. But meh, I'll 11057 let the user define that, D only allows one empty arg thing anyway 11058 so the method UDAs are irrelevant. 11059 +/ 11060 if(callFunction) 11061 +/ 11062 11063 auto format = cgi.request("format", defaultFormat!overload()); 11064 auto wantsFormFormat = format.startsWith("form-"); 11065 11066 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 11067 // Should I still show the form on a json thing? idk... 11068 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 11069 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 11070 return true; 11071 } 11072 11073 try { 11074 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 11075 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 11076 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 11077 } catch(Throwable t) { 11078 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 11079 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 11080 } 11081 return true; 11082 //}} 11083 11084 //cgi.header("Accept: POST"); // FIXME list the real thing 11085 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 11086 //return true; 11087 } 11088 } 11089 } 11090 case "GET script.js": 11091 cgi.setResponseContentType("text/javascript"); 11092 cgi.gzipResponse = true; 11093 cgi.write(presenter.script(), true); 11094 return true; 11095 case "GET style.css": 11096 cgi.setResponseContentType("text/css"); 11097 cgi.gzipResponse = true; 11098 cgi.write(presenter.style(), true); 11099 return true; 11100 default: 11101 return false; 11102 } 11103 11104 assert(0); 11105 } 11106 return DispatcherDefinition!internalHandler(urlPrefix, false); 11107 } 11108 11109 string defaultFormat(alias method)() { 11110 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 11111 foreach(attr; __traits(getAttributes, method)) { 11112 static if(is(typeof(attr) == DefaultFormat)) { 11113 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 11114 return attr.value; 11115 } 11116 } 11117 return "html"; 11118 } 11119 11120 struct Paginated(T) { 11121 T[] items; 11122 string nextPageUrl; 11123 } 11124 11125 template urlNamesForMethod(alias method, string default_) { 11126 string[] helper() { 11127 auto verb = Cgi.RequestMethod.GET; 11128 bool foundVerb = false; 11129 bool foundNoun = false; 11130 11131 string def = default_; 11132 11133 bool hasAutomaticForm = false; 11134 11135 foreach(attr; __traits(getAttributes, method)) { 11136 static if(is(typeof(attr) == Cgi.RequestMethod)) { 11137 verb = attr; 11138 if(foundVerb) 11139 assert(0, "Multiple http verbs on one function is not currently supported"); 11140 foundVerb = true; 11141 } 11142 static if(is(typeof(attr) == UrlName)) { 11143 if(foundNoun) 11144 assert(0, "Multiple url names on one function is not currently supported"); 11145 foundNoun = true; 11146 def = attr.name; 11147 } 11148 static if(__traits(isSame, attr, AutomaticForm)) { 11149 hasAutomaticForm = true; 11150 } 11151 } 11152 11153 if(def is null) 11154 def = "__null"; 11155 11156 string[] ret; 11157 11158 static if(is(typeof(method) R == return)) { 11159 static if(is(R : WebObject)) { 11160 def ~= "/"; 11161 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 11162 ret ~= v ~ " " ~ def; 11163 } else { 11164 if(hasAutomaticForm) { 11165 ret ~= "GET " ~ def; 11166 ret ~= "POST " ~ def; 11167 } else { 11168 ret ~= to!string(verb) ~ " " ~ def; 11169 } 11170 } 11171 } else static assert(0); 11172 11173 return ret; 11174 } 11175 enum urlNamesForMethod = helper(); 11176 } 11177 11178 11179 enum AccessCheck { 11180 allowed, 11181 denied, 11182 nonExistant, 11183 } 11184 11185 enum Operation { 11186 show, 11187 create, 11188 replace, 11189 remove, 11190 update 11191 } 11192 11193 enum UpdateResult { 11194 accessDenied, 11195 noSuchResource, 11196 success, 11197 failure, 11198 unnecessary 11199 } 11200 11201 enum ValidationResult { 11202 valid, 11203 invalid 11204 } 11205 11206 11207 /++ 11208 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 11209 11210 WARNING: this is not stable. 11211 +/ 11212 class RestObject(CRTP) : WebObject { 11213 11214 import arsd.dom; 11215 import arsd.jsvar; 11216 11217 /// Prepare the object to be shown. 11218 void show() {} 11219 /// ditto 11220 void show(string urlId) { 11221 load(urlId); 11222 show(); 11223 } 11224 11225 /// Override this to provide access control to this object. 11226 AccessCheck accessCheck(string urlId, Operation operation) { 11227 return AccessCheck.allowed; 11228 } 11229 11230 ValidationResult validate() { 11231 // FIXME 11232 return ValidationResult.valid; 11233 } 11234 11235 string getUrlSlug() { 11236 import std.conv; 11237 static if(is(typeof(CRTP.id))) 11238 return to!string((cast(CRTP) this).id); 11239 else 11240 return null; 11241 } 11242 11243 // The functions with more arguments are the low-level ones, 11244 // they forward to the ones with fewer arguments by default. 11245 11246 // POST on a parent collection - this is called from a collection class after the members are updated 11247 /++ 11248 Given a populated object, this creates a new entry. Returns the url identifier 11249 of the new object. 11250 +/ 11251 string create(scope void delegate() applyChanges) { 11252 applyChanges(); 11253 save(); 11254 return getUrlSlug(); 11255 } 11256 11257 void replace() { 11258 save(); 11259 } 11260 void replace(string urlId, scope void delegate() applyChanges) { 11261 load(urlId); 11262 applyChanges(); 11263 replace(); 11264 } 11265 11266 void update(string[] fieldList) { 11267 save(); 11268 } 11269 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 11270 load(urlId); 11271 applyChanges(); 11272 update(fieldList); 11273 } 11274 11275 void remove() {} 11276 11277 void remove(string urlId) { 11278 load(urlId); 11279 remove(); 11280 } 11281 11282 abstract void load(string urlId); 11283 abstract void save(); 11284 11285 Element toHtml(Presenter)(Presenter presenter) { 11286 import arsd.dom; 11287 import std.conv; 11288 auto obj = cast(CRTP) this; 11289 auto div = Element.make("div"); 11290 div.addClass("Dclass_" ~ CRTP.stringof); 11291 div.dataset.url = getUrlSlug(); 11292 bool first = true; 11293 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11294 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11295 if(!first) div.addChild("br"); else first = false; 11296 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11297 } 11298 return div; 11299 } 11300 11301 var toJson() { 11302 import arsd.jsvar; 11303 var v = var.emptyObject(); 11304 auto obj = cast(CRTP) this; 11305 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11306 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11307 v[memberName] = __traits(getMember, obj, memberName); 11308 } 11309 return v; 11310 } 11311 11312 /+ 11313 auto structOf(this This) { 11314 11315 } 11316 +/ 11317 } 11318 11319 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11320 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11321 11322 /++ 11323 Base class for REST collections. 11324 +/ 11325 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11326 /// You might subclass this and use the cgi object's query params 11327 /// to implement a search filter, for example. 11328 /// 11329 /// FIXME: design a way to auto-generate that form 11330 /// (other than using the WebObject thing above lol 11331 // it'll prolly just be some searchParams UDA or maybe an enum. 11332 // 11333 // pagination too perhaps. 11334 // 11335 // and sorting too 11336 IndexResult index() { return IndexResult.init; } 11337 11338 string[] sortableFields() { return null; } 11339 string[] searchableFields() { return null; } 11340 11341 struct IndexResult { 11342 Obj[] results; 11343 11344 string[] sortableFields; 11345 11346 string previousPageIdentifier; 11347 string nextPageIdentifier; 11348 string firstPageIdentifier; 11349 string lastPageIdentifier; 11350 11351 int numberOfPages; 11352 } 11353 11354 override string create(scope void delegate() applyChanges) { assert(0); } 11355 override void load(string urlId) { assert(0); } 11356 override void save() { assert(0); } 11357 override void show() { 11358 index(); 11359 } 11360 override void show(string urlId) { 11361 show(); 11362 } 11363 11364 /// Proxy POST requests (create calls) to the child collection 11365 alias PostProxy = Obj; 11366 } 11367 11368 /++ 11369 Serves a REST object, similar to a Ruby on Rails resource. 11370 11371 You put data members in your class. cgi.d will automatically make something out of those. 11372 11373 It will call your constructor with the ID from the URL. This may be null. 11374 It will then populate the data members from the request. 11375 It will then call a method, if present, telling what happened. You don't need to write these! 11376 It finally returns a reply. 11377 11378 Your methods are passed a list of fields it actually set. 11379 11380 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11381 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11382 with relative linking. But meh.) 11383 11384 GET /items -> index. all values not set. 11385 GET /items/id -> get. only ID will be set, other params ignored. 11386 POST /items -> create. values set as given 11387 PUT /items/id -> replace. values set as given 11388 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11389 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11390 PATCH /items/id -> update. values set as given, list of changed fields passed 11391 or POST /items/id with cgi.post["_method"] == "PATCH" 11392 DELETE /items/id -> destroy. only ID guaranteed to be set 11393 or POST /items/id with cgi.post["_method"] == "DELETE" 11394 11395 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11396 redirect you away from it. 11397 11398 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11399 11400 I will also let you change the default, if you must. 11401 11402 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11403 11404 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11405 that follow the same thing. They may be individual resources or collections themselves. 11406 11407 Your class is expected to have at least the following methods: 11408 11409 FIXME: i kinda wanna add a routes object to the initialize call 11410 11411 create 11412 Create returns the new address on success, some code on failure. 11413 show 11414 index 11415 update 11416 remove 11417 11418 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11419 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11420 11421 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11422 11423 NOT IMPLEMENTED 11424 11425 11426 Really, a collection is a resource with a bunch of subresources. 11427 11428 GET /items 11429 index because it is GET on the top resource 11430 11431 GET /items/foo 11432 item but different than items? 11433 11434 class Items { 11435 11436 } 11437 11438 ... but meh, a collection can be automated. not worth making it 11439 a separate thing, let's look at a real example. Users has many 11440 items and a virtual one, /users/current. 11441 11442 the individual users have properties and two sub-resources: 11443 session, which is just one, and comments, a collection. 11444 11445 class User : RestObject!() { // no parent 11446 int id; 11447 string name; 11448 11449 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11450 // but you can override them to do it differently. 11451 11452 // any member which is of type RestObject can be linked automatically via href btw. 11453 11454 void show() {} 11455 void show(string urlId) {} // automated! GET of this specific thing 11456 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11457 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11458 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11459 void remove(string urlId) {} // DELETE 11460 11461 void load(string urlId) {} // the default implementation of show() populates the id, then 11462 11463 this() {} 11464 11465 mixin Subresource!Session; 11466 mixin Subresource!Comment; 11467 } 11468 11469 class Session : RestObject!() { 11470 // the parent object may not be fully constructed/loaded 11471 this(User parent) {} 11472 11473 } 11474 11475 class Comment : CollectionOf!Comment { 11476 this(User parent) {} 11477 } 11478 11479 class Users : CollectionOf!User { 11480 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11481 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11482 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11483 } 11484 11485 +/ 11486 auto serveRestObject(T)(string urlPrefix) { 11487 assert(urlPrefix[0] == '/'); 11488 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11489 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11490 string url = cgi.pathInfo[urlPrefix.length .. $]; 11491 11492 if(url.length && url[$ - 1] == '/') { 11493 // remove the final slash... 11494 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11495 return true; 11496 } 11497 11498 return restObjectServeHandler!T(cgi, presenter, url); 11499 } 11500 return DispatcherDefinition!internalHandler(urlPrefix, false); 11501 } 11502 11503 /+ 11504 /// Convenience method for serving a collection. It will be named the same 11505 /// as type T, just with an s at the end. If you need any further, just 11506 /// write the class yourself. 11507 auto serveRestCollectionOf(T)(string urlPrefix) { 11508 assert(urlPrefix[0] == '/'); 11509 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11510 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11511 } 11512 +/ 11513 11514 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11515 string urlId = null; 11516 if(url.length && url[0] == '/') { 11517 // asking for a subobject 11518 urlId = url[1 .. $]; 11519 foreach(idx, ch; urlId) { 11520 if(ch == '/') { 11521 urlId = urlId[0 .. idx]; 11522 break; 11523 } 11524 } 11525 } 11526 11527 // FIXME handle other subresources 11528 11529 static if(is(T : CollectionOf!(C), C)) { 11530 if(urlId !is null) { 11531 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11532 } 11533 } 11534 11535 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11536 11537 auto obj = new T(); 11538 obj.initialize(cgi); 11539 // FIXME: populate reflection info delegates 11540 11541 11542 // FIXME: I am not happy with this. 11543 switch(urlId) { 11544 case "script.js": 11545 cgi.setResponseContentType("text/javascript"); 11546 cgi.gzipResponse = true; 11547 cgi.write(presenter.script(), true); 11548 return true; 11549 case "style.css": 11550 cgi.setResponseContentType("text/css"); 11551 cgi.gzipResponse = true; 11552 cgi.write(presenter.style(), true); 11553 return true; 11554 default: 11555 // intentionally blank 11556 } 11557 11558 11559 11560 11561 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11562 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11563 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11564 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11565 } 11566 } 11567 void applyChanges() { 11568 applyChangesTemplate(cgi, obj); 11569 } 11570 11571 string[] modifiedList; 11572 11573 void writeObject(bool addFormLinks) { 11574 if(cgi.request("format") == "json") { 11575 cgi.setResponseContentType("application/json"); 11576 cgi.write(obj.toJson().toString, true); 11577 } else { 11578 auto container = presenter.htmlContainer(); 11579 if(addFormLinks) { 11580 static if(is(T : CollectionOf!(C), C)) 11581 container.appendHtml(` 11582 <form> 11583 <button type="submit" name="_method" value="POST">Create New</button> 11584 </form> 11585 `); 11586 else 11587 container.appendHtml(` 11588 <a href="..">Back</a> 11589 <form> 11590 <button type="submit" name="_method" value="PATCH">Edit</button> 11591 <button type="submit" name="_method" value="DELETE">Delete</button> 11592 </form> 11593 `); 11594 } 11595 container.appendChild(obj.toHtml(presenter)); 11596 cgi.write(container.parentDocument.toString, true); 11597 } 11598 } 11599 11600 // FIXME: I think I need a set type in here.... 11601 // it will be nice to pass sets of members. 11602 11603 try 11604 switch(cgi.requestMethod) { 11605 case Cgi.RequestMethod.GET: 11606 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11607 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11608 11609 // automatic forms here for usable basic auto site from browser. 11610 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11611 switch(cgi.request("_method", "GET")) { 11612 case "GET": 11613 static if(is(T : CollectionOf!(C), C)) { 11614 auto results = obj.index(); 11615 if(cgi.request("format", "html") == "html") { 11616 auto container = presenter.htmlContainer(); 11617 auto html = presenter.formatReturnValueAsHtml(results.results); 11618 container.appendHtml(` 11619 <form> 11620 <button type="submit" name="_method" value="POST">Create New</button> 11621 </form> 11622 `); 11623 11624 container.appendChild(html); 11625 cgi.write(container.parentDocument.toString, true); 11626 } else { 11627 cgi.setResponseContentType("application/json"); 11628 import arsd.jsvar; 11629 var json = var.emptyArray; 11630 foreach(r; results.results) { 11631 var o = var.emptyObject; 11632 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11633 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11634 o[memberName] = __traits(getMember, r, memberName); 11635 } 11636 11637 json ~= o; 11638 } 11639 cgi.write(json.toJson(), true); 11640 } 11641 } else { 11642 obj.show(urlId); 11643 writeObject(true); 11644 } 11645 break; 11646 case "PATCH": 11647 obj.load(urlId); 11648 goto case; 11649 case "PUT": 11650 case "POST": 11651 // an editing form for the object 11652 auto container = presenter.htmlContainer(); 11653 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11654 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11655 } else { 11656 auto form = presenter.createAutomaticFormForObject(obj); 11657 } 11658 form.attrs.method = "POST"; 11659 form.setValue("_method", cgi.request("_method", "GET")); 11660 container.appendChild(form); 11661 cgi.write(container.parentDocument.toString(), true); 11662 break; 11663 case "DELETE": 11664 // FIXME: a delete form for the object (can be phrased "are you sure?") 11665 auto container = presenter.htmlContainer(); 11666 container.appendHtml(` 11667 <form method="POST"> 11668 Are you sure you want to delete this item? 11669 <input type="hidden" name="_method" value="DELETE" /> 11670 <input type="submit" value="Yes, Delete It" /> 11671 </form> 11672 11673 `); 11674 cgi.write(container.parentDocument.toString(), true); 11675 break; 11676 default: 11677 cgi.write("bad method\n", true); 11678 } 11679 break; 11680 case Cgi.RequestMethod.POST: 11681 // this is to allow compatibility with HTML forms 11682 switch(cgi.request("_method", "POST")) { 11683 case "PUT": 11684 goto PUT; 11685 case "PATCH": 11686 goto PATCH; 11687 case "DELETE": 11688 goto DELETE; 11689 case "POST": 11690 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11691 auto p = new obj.PostProxy(); 11692 void specialApplyChanges() { 11693 applyChangesTemplate(cgi, p); 11694 } 11695 string n = p.create(&specialApplyChanges); 11696 } else { 11697 string n = obj.create(&applyChanges); 11698 } 11699 11700 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11701 cgi.setResponseLocation(newUrl); 11702 cgi.setResponseStatus("201 Created"); 11703 cgi.write(`The object has been created.`); 11704 break; 11705 default: 11706 cgi.write("bad method\n", true); 11707 } 11708 // FIXME this should be valid on the collection, but not the child.... 11709 // 303 See Other 11710 break; 11711 case Cgi.RequestMethod.PUT: 11712 PUT: 11713 obj.replace(urlId, &applyChanges); 11714 writeObject(false); 11715 break; 11716 case Cgi.RequestMethod.PATCH: 11717 PATCH: 11718 obj.update(urlId, &applyChanges, modifiedList); 11719 writeObject(false); 11720 break; 11721 case Cgi.RequestMethod.DELETE: 11722 DELETE: 11723 obj.remove(urlId); 11724 cgi.setResponseStatus("204 No Content"); 11725 break; 11726 default: 11727 // FIXME: OPTIONS, HEAD 11728 } 11729 catch(Throwable t) { 11730 presenter.presentExceptionAsHtml(cgi, t); 11731 } 11732 11733 return true; 11734 } 11735 11736 /+ 11737 struct SetOfFields(T) { 11738 private void[0][string] storage; 11739 void set(string what) { 11740 //storage[what] = 11741 } 11742 void unset(string what) {} 11743 void setAll() {} 11744 void unsetAll() {} 11745 bool isPresent(string what) { return false; } 11746 } 11747 +/ 11748 11749 /+ 11750 enum readonly; 11751 enum hideonindex; 11752 +/ 11753 11754 /++ 11755 Returns true if I recommend gzipping content of this type. You might 11756 want to call it from your Presenter classes before calling cgi.write. 11757 11758 --- 11759 cgi.setResponseContentType(yourContentType); 11760 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11761 cgi.write(yourData, true); 11762 --- 11763 11764 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11765 11766 11767 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11768 11769 History: 11770 Added January 28, 2023 (dub v11.0) 11771 +/ 11772 bool gzipRecommendedForContentType(string contentType) { 11773 if(contentType.startsWith("text/")) 11774 return true; 11775 if(contentType.startsWith("application/javascript")) 11776 return true; 11777 11778 return false; 11779 } 11780 11781 /++ 11782 Serves a static file. To be used with [dispatcher]. 11783 11784 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11785 +/ 11786 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11787 // https://baus.net/on-tcp_cork/ 11788 // man 2 sendfile 11789 assert(urlPrefix[0] == '/'); 11790 if(filename is null) 11791 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11792 if(contentType is null) { 11793 contentType = contentTypeFromFileExtension(filename); 11794 } 11795 11796 static struct DispatcherDetails { 11797 string filename; 11798 string contentType; 11799 } 11800 11801 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11802 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11803 cgi.setCache(true); 11804 cgi.setResponseContentType(details.contentType); 11805 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11806 cgi.write(std.file.read(details.filename), true); 11807 return true; 11808 } 11809 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11810 } 11811 11812 /++ 11813 Serves static data. To be used with [dispatcher]. 11814 11815 History: 11816 Added October 31, 2021 11817 +/ 11818 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11819 assert(urlPrefix[0] == '/'); 11820 if(contentType is null) { 11821 contentType = contentTypeFromFileExtension(urlPrefix); 11822 } 11823 11824 static struct DispatcherDetails { 11825 immutable(void)[] data; 11826 string contentType; 11827 } 11828 11829 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11830 cgi.setCache(true); 11831 cgi.setResponseContentType(details.contentType); 11832 cgi.write(details.data, true); 11833 return true; 11834 } 11835 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11836 } 11837 11838 string contentTypeFromFileExtension(string filename) { 11839 if(filename.endsWith(".png")) 11840 return "image/png"; 11841 if(filename.endsWith(".apng")) 11842 return "image/apng"; 11843 if(filename.endsWith(".svg")) 11844 return "image/svg+xml"; 11845 if(filename.endsWith(".jpg")) 11846 return "image/jpeg"; 11847 if(filename.endsWith(".html")) 11848 return "text/html"; 11849 if(filename.endsWith(".css")) 11850 return "text/css"; 11851 if(filename.endsWith(".js")) 11852 return "application/javascript"; 11853 if(filename.endsWith(".wasm")) 11854 return "application/wasm"; 11855 if(filename.endsWith(".mp3")) 11856 return "audio/mpeg"; 11857 if(filename.endsWith(".pdf")) 11858 return "application/pdf"; 11859 return null; 11860 } 11861 11862 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11863 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11864 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11865 assert(urlPrefix[0] == '/'); 11866 assert(urlPrefix[$-1] == '/'); 11867 11868 static struct DispatcherDetails { 11869 string directory; 11870 bool recursive; 11871 } 11872 11873 if(directory is null) 11874 directory = urlPrefix[1 .. $]; 11875 11876 if(directory.length == 0) 11877 directory = "./"; 11878 11879 assert(directory[$-1] == '/'); 11880 11881 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11882 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11883 11884 if(details.recursive) { 11885 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11886 if(file.indexOf("\\") != -1) 11887 return false; 11888 11889 import std.path; 11890 11891 file = std.path.buildNormalizedPath(file); 11892 enum upOneDir = ".." ~ std.path.dirSeparator; 11893 11894 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11895 if(file == ".." || file.startsWith(upOneDir)) 11896 return false; 11897 if(std.path.isAbsolute(file)) 11898 return false; 11899 11900 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11901 11902 // once it passes these filters it is probably ok. 11903 } else { 11904 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11905 return false; 11906 } 11907 11908 if(file.length == 0) 11909 return false; 11910 11911 auto contentType = contentTypeFromFileExtension(file); 11912 11913 auto fn = details.directory ~ file; 11914 if(std.file.exists(fn)) { 11915 //if(contentType.indexOf("image/") == 0) 11916 //cgi.setCache(true); 11917 //else if(contentType.indexOf("audio/") == 0) 11918 cgi.setCache(true); 11919 cgi.setResponseContentType(contentType); 11920 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11921 cgi.write(std.file.read(fn), true); 11922 return true; 11923 } else { 11924 return false; 11925 } 11926 } 11927 11928 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11929 } 11930 11931 /++ 11932 Redirects one url to another 11933 11934 See_Also: [dispatcher], [serveStaticFile] 11935 +/ 11936 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11937 assert(urlPrefix[0] == '/'); 11938 static struct DispatcherDetails { 11939 string redirectTo; 11940 string code; 11941 } 11942 11943 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11944 cgi.setResponseLocation(details.redirectTo, true, details.code); 11945 return true; 11946 } 11947 11948 11949 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11950 } 11951 11952 /// Used exclusively with `dispatchTo` 11953 struct DispatcherData(Presenter) { 11954 Cgi cgi; /// You can use this cgi object. 11955 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11956 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11957 } 11958 11959 /++ 11960 Dispatches the URL to a specific function. 11961 +/ 11962 auto handleWith(alias handler)(string urlPrefix) { 11963 // cuz I'm too lazy to do it better right now 11964 static class Hack : WebObject { 11965 static import std.traits; 11966 @UrlName("") 11967 auto handle(std.traits.Parameters!handler args) { 11968 return handler(args); 11969 } 11970 } 11971 11972 return urlPrefix.serveApiInternal!Hack; 11973 } 11974 11975 /++ 11976 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11977 11978 --- 11979 bool other(DD)(DD dd) { 11980 return dd.dispatcher!( 11981 "/whatever".serveRedirect("/success"), 11982 "/api/".serveApi!MyClass 11983 ); 11984 } 11985 --- 11986 11987 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11988 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11989 Or, of course, you could just use the exact type in your own code. 11990 11991 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11992 good job. 11993 11994 11995 +/ 11996 auto dispatchTo(alias handler)(string urlPrefix) { 11997 assert(urlPrefix[0] == '/'); 11998 assert(urlPrefix[$-1] != '/'); 11999 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 12000 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 12001 } 12002 12003 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 12004 } 12005 12006 /++ 12007 See [serveStaticFile] if you want to serve a file off disk. 12008 12009 History: 12010 Added January 28, 2023 (dub v11.0) 12011 +/ 12012 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 12013 assert(urlPrefix[0] == '/'); 12014 12015 static struct DispatcherDetails { 12016 immutable(ubyte)[] data; 12017 string contentType; 12018 string filenameToSuggestAsDownload; 12019 } 12020 12021 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 12022 cgi.setCache(true); 12023 cgi.setResponseContentType(details.contentType); 12024 if(details.filenameToSuggestAsDownload.length) 12025 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 12026 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 12027 cgi.write(details.data, true); 12028 return true; 12029 } 12030 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 12031 } 12032 12033 /++ 12034 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 12035 12036 History: 12037 Added January 28, 2023 (dub v11.0) 12038 +/ 12039 alias KeepExistingPresenter = typeof(null); 12040 12041 /++ 12042 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 12043 this issues the given errorCode and stops processing. 12044 12045 --- 12046 bool hasAdminPermissions(Cgi cgi) { 12047 return true; 12048 } 12049 12050 mixin DispatcherMain!( 12051 "/admin".dispatchSubsection!( 12052 passFilterOrIssueError!(hasAdminPermissions, 403), 12053 KeepExistingPresenter, 12054 "/".serveApi!AdminFunctions 12055 ) 12056 ); 12057 --- 12058 12059 History: 12060 Added January 28, 2023 (dub v11.0) 12061 +/ 12062 template passFilterOrIssueError(alias filter, int errorCode) { 12063 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 12064 if(filter(dd.cgi)) 12065 return true; 12066 dd.presenter.renderBasicError(dd.cgi, errorCode); 12067 return false; 12068 } 12069 } 12070 12071 /++ 12072 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 12073 and then be dispatched to their own handlers. 12074 12075 --- 12076 /+ 12077 // a long-form filter function 12078 bool permissionCheck(DispatcherData)(DispatcherData dd) { 12079 // you are permitted to call mutable methods on the Cgi object 12080 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 12081 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 12082 12083 if(checkPermissionOnRequest(dd.cgi)) { 12084 return true; // OK, allow processing to continue 12085 } else { 12086 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 12087 return false; // and stop further processing into this subsection 12088 } 12089 } 12090 +/ 12091 12092 // but you can also do short-form filters: 12093 12094 bool permissionCheck(Cgi cgi) { 12095 return ("ok" in cgi.get) !is null; 12096 } 12097 12098 // handler for the subsection 12099 class AdminClass : WebObject { 12100 int foo() { return 5; } 12101 } 12102 12103 // handler for the main site 12104 class TheMainSite : WebObject {} 12105 12106 mixin DispatcherMain!( 12107 "/admin".dispatchSubsection!( 12108 // converts our short-form filter into a long-form filter 12109 passFilterOrIssueError!(permissionCheck, 403), 12110 // can use a new presenter if wanted for the subsection 12111 KeepExistingPresenter, 12112 // and then provide child route dispatchers 12113 "/".serveApi!AdminClass 12114 ), 12115 // and back to the top level 12116 "/".serveApi!TheMainSite 12117 ); 12118 --- 12119 12120 Note you can encapsulate sections in files like this: 12121 12122 --- 12123 auto adminDispatcher(string urlPrefix) { 12124 return urlPrefix.dispatchSubsection!( 12125 .... 12126 ); 12127 } 12128 12129 mixin DispatcherMain!( 12130 "/admin".adminDispatcher, 12131 // and so on 12132 ) 12133 --- 12134 12135 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 12136 12137 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 12138 12139 12140 History: 12141 Added January 28, 2023 (dub v11.0) 12142 +/ 12143 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 12144 assert(urlPrefix[0] == '/'); 12145 assert(urlPrefix[$-1] != '/'); 12146 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 12147 static if(!is(PreRequestFilter == typeof(null))) { 12148 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 12149 return true; // we handled it by rejecting it 12150 } 12151 12152 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 12153 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 12154 } else { 12155 auto newPresenter = new NewPresenter(); 12156 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 12157 } 12158 } 12159 12160 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 12161 } 12162 12163 /++ 12164 A URL dispatcher. 12165 12166 --- 12167 if(cgi.dispatcher!( 12168 "/api/".serveApi!MyApiClass, 12169 "/objects/lol".serveRestObject!MyRestObject, 12170 "/file.js".serveStaticFile, 12171 "/admin/".dispatchTo!adminHandler 12172 )) return; 12173 --- 12174 12175 12176 You define a series of url prefixes followed by handlers. 12177 12178 You may want to do different pre- and post- processing there, for example, 12179 an authorization check and different page layout. You can use different 12180 presenters and different function chains. See [dispatchSubsection] for details. 12181 12182 [dispatchTo] will send the request to another function for handling. 12183 +/ 12184 template dispatcher(definitions...) { 12185 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 12186 static if(is(Presenter == typeof(null))) { 12187 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 12188 auto presenter = new GenericWebPresenter(); 12189 } else 12190 alias presenter = presenterArg; 12191 12192 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 12193 } 12194 12195 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 12196 // I can prolly make this more efficient later but meh. 12197 foreach(definition; definitions) { 12198 if(definition.rejectFurther) { 12199 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 12200 auto ret = definition.handler( 12201 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 12202 dispatcherData.cgi, dispatcherData.presenter, definition.details); 12203 if(ret) 12204 return true; 12205 } 12206 } else if( 12207 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 12208 // cgi.d dispatcher urls must be complete or have a /; 12209 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 12210 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 12211 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 12212 ) { 12213 auto ret = definition.handler( 12214 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 12215 dispatcherData.cgi, dispatcherData.presenter, definition.details); 12216 if(ret) 12217 return true; 12218 } 12219 } 12220 return false; 12221 } 12222 } 12223 12224 }); 12225 12226 private struct StackBuffer { 12227 char[1024] initial = void; 12228 char[] buffer; 12229 size_t position; 12230 12231 this(int a) { 12232 buffer = initial[]; 12233 position = 0; 12234 } 12235 12236 void add(in char[] what) { 12237 if(position + what.length > buffer.length) 12238 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 12239 buffer[position .. position + what.length] = what[]; 12240 position += what.length; 12241 } 12242 12243 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 12244 add(w1); 12245 add(w2); 12246 add(w3); 12247 } 12248 12249 void add(long v) { 12250 char[16] buffer = void; 12251 auto pos = buffer.length; 12252 bool negative; 12253 if(v < 0) { 12254 negative = true; 12255 v = -v; 12256 } 12257 do { 12258 buffer[--pos] = cast(char) (v % 10 + '0'); 12259 v /= 10; 12260 } while(v); 12261 12262 if(negative) 12263 buffer[--pos] = '-'; 12264 12265 auto res = buffer[pos .. $]; 12266 12267 add(res[]); 12268 } 12269 12270 char[] get() @nogc { 12271 return buffer[0 .. position]; 12272 } 12273 } 12274 12275 // duplicated in http2.d 12276 private static string getHttpCodeText(int code) pure nothrow @nogc { 12277 switch(code) { 12278 case 200: return "200 OK"; 12279 case 201: return "201 Created"; 12280 case 202: return "202 Accepted"; 12281 case 203: return "203 Non-Authoritative Information"; 12282 case 204: return "204 No Content"; 12283 case 205: return "205 Reset Content"; 12284 case 206: return "206 Partial Content"; 12285 // 12286 case 300: return "300 Multiple Choices"; 12287 case 301: return "301 Moved Permanently"; 12288 case 302: return "302 Found"; 12289 case 303: return "303 See Other"; 12290 case 304: return "304 Not Modified"; 12291 case 305: return "305 Use Proxy"; 12292 case 307: return "307 Temporary Redirect"; 12293 case 308: return "308 Permanent Redirect"; 12294 12295 // 12296 case 400: return "400 Bad Request"; 12297 case 401: return "401 Unauthorized"; 12298 case 402: return "402 Payment Required"; 12299 case 403: return "403 Forbidden"; 12300 case 404: return "404 Not Found"; 12301 case 405: return "405 Method Not Allowed"; 12302 case 406: return "406 Not Acceptable"; 12303 case 407: return "407 Proxy Authentication Required"; 12304 case 408: return "408 Request Timeout"; 12305 case 409: return "409 Conflict"; 12306 case 410: return "410 Gone"; 12307 case 411: return "411 Length Required"; 12308 case 412: return "412 Precondition Failed"; 12309 case 413: return "413 Payload Too Large"; 12310 case 414: return "414 URI Too Long"; 12311 case 415: return "415 Unsupported Media Type"; 12312 case 416: return "416 Range Not Satisfiable"; 12313 case 417: return "417 Expectation Failed"; 12314 case 418: return "418 I'm a teapot"; 12315 case 421: return "421 Misdirected Request"; 12316 case 422: return "422 Unprocessable Entity (WebDAV)"; 12317 case 423: return "423 Locked (WebDAV)"; 12318 case 424: return "424 Failed Dependency (WebDAV)"; 12319 case 425: return "425 Too Early"; 12320 case 426: return "426 Upgrade Required"; 12321 case 428: return "428 Precondition Required"; 12322 case 431: return "431 Request Header Fields Too Large"; 12323 case 451: return "451 Unavailable For Legal Reasons"; 12324 12325 case 500: return "500 Internal Server Error"; 12326 case 501: return "501 Not Implemented"; 12327 case 502: return "502 Bad Gateway"; 12328 case 503: return "503 Service Unavailable"; 12329 case 504: return "504 Gateway Timeout"; 12330 case 505: return "505 HTTP Version Not Supported"; 12331 case 506: return "506 Variant Also Negotiates"; 12332 case 507: return "507 Insufficient Storage (WebDAV)"; 12333 case 508: return "508 Loop Detected (WebDAV)"; 12334 case 510: return "510 Not Extended"; 12335 case 511: return "511 Network Authentication Required"; 12336 // 12337 default: assert(0, "Unsupported http code"); 12338 } 12339 } 12340 12341 12342 /+ 12343 /++ 12344 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12345 12346 It relies on jsvar.d and dom.d. 12347 12348 12349 You can get javascript out of it to call. The generated functions need to look 12350 like 12351 12352 function name(a,b,c,d,e) { 12353 return _call("name", {"realName":a,"sds":b}); 12354 } 12355 12356 And _call returns an object you can call or set up or whatever. 12357 +/ 12358 bool apiDispatcher()(Cgi cgi) { 12359 import arsd.jsvar; 12360 import arsd.dom; 12361 } 12362 +/ 12363 version(linux) 12364 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12365 /* 12366 Copyright: Adam D. Ruppe, 2008 - 2023 12367 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12368 Authors: Adam D. Ruppe 12369 12370 Copyright Adam D. Ruppe 2008 - 2023. 12371 Distributed under the Boost Software License, Version 1.0. 12372 (See accompanying file LICENSE_1_0.txt or copy at 12373 http://www.boost.org/LICENSE_1_0.txt) 12374 */