1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. Offers both lower- and higher- level api options among other common (optional) things like websocket and event source serving support, session management, and job scheduling. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Or: 78 --- 79 import arsd.cgi; 80 81 class MyApi : WebObject { 82 @UrlName("") 83 string hello(string name = null) { 84 if(name is null) 85 return "Hello, world!"; 86 else 87 return "Hello, " ~ name; 88 } 89 } 90 mixin DispatcherMain!( 91 "/".serveApi!MyApi 92 ); 93 --- 94 95 $(NOTE 96 Please note that using the higher-level api will add a dependency on arsd.dom and arsd.jsvar to your application. 97 If you use `dmd -i` or `ldc2 -i` to build, it will just work, but with dub, you will have do `dub add arsd-official:jsvar` 98 and `dub add arsd-official:dom` yourself. 99 ) 100 101 Test on console (works in any interface mode): 102 $(CONSOLE 103 $ ./cgi_hello GET / name=whatever 104 ) 105 106 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 107 $(CONSOLE 108 $ ./cgi_hello --port 8080 109 # now you can go to http://localhost:8080/?name=whatever 110 ) 111 112 Please note: the default port for http is 8085 and for scgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to code your own with [RequestServer], however. 113 114 115 Build_Configurations: 116 117 cgi.d tries to be flexible to meet your needs. It is possible to configure it both at runtime (by writing your own `main` function and constructing a [RequestServer] object) or at compile time using the `version` switch to the compiler or a dub `subConfiguration`. 118 119 If you are using `dub`, use: 120 121 ```sdlang 122 subConfiguration "arsd-official:cgi" "VALUE_HERE" 123 ``` 124 125 or to dub.json: 126 127 ```json 128 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 129 ``` 130 131 to change versions. The possible options for `VALUE_HERE` are: 132 133 $(LIST 134 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default for dub builds. You can run the program then connect directly to it from your browser. Note: prior to version 11, this would be embedded_httpd_processes on Linux and embedded_httpd_threads everywhere else. It now means embedded_httpd_hybrid everywhere supported and embedded_httpd_threads everywhere else. 135 * `cgi` for traditional cgi binaries. These are run by an outside web server as-needed to handle requests. 136 * `fastcgi` for FastCGI builds. FastCGI is managed from an outside helper, there's one built into Microsoft IIS, Apache httpd, and Lighttpd, and a generic program you can use with nginx called `spawn-fcgi`. If you don't already know how to use it, I suggest you use one of the other modes. 137 * `scgi` for SCGI builds. SCGI is a simplified form of FastCGI, where you run the server as an application service which is proxied by your outside webserver. 138 * `stdio_http` for speaking raw http over stdin and stdout. This is made for systemd services. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 139 ) 140 141 With dmd, use: 142 143 $(TABLE_ROWS 144 145 * + Interfaces 146 + (mutually exclusive) 147 148 * - `-version=plain_cgi` 149 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 150 * - `-version=embedded_httpd` 151 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 152 * - `-version=fastcgi` 153 - A FastCGI executable will be generated. 154 * - `-version=scgi` 155 - A SCGI (SimpleCGI) executable will be generated. 156 * - `-version=embedded_httpd_hybrid` 157 - A HTTP server that uses a combination of processes, threads, and fibers to better handle large numbers of idle connections. Recommended if you are going to serve websockets in a non-local application. 158 * - `-version=embedded_httpd_threads` 159 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 160 * - `-version=embedded_httpd_processes` 161 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 162 * - `-version=embedded_httpd_processes_accept_after_fork` 163 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 164 * - `-version=stdio_http` 165 - The embedded HTTP server will be spoken over stdin and stdout. 166 167 * + Tweaks 168 + (can be used together with others) 169 170 * - `-version=cgi_with_websocket` 171 - The CGI class has websocket server support. (This is on by default now.) 172 173 * - `-version=with_openssl` 174 - not currently used 175 * - `-version=cgi_embedded_sessions` 176 - The session server will be embedded in the cgi.d server process 177 * - `-version=cgi_session_server_process` 178 - The session will be provided in a separate process, provided by cgi.d. 179 ) 180 181 For example, 182 183 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 184 185 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 186 187 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 188 189 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 190 191 Simulating_requests: 192 193 If you are using one of the [GenericMain] or [DispatcherMain] mixins, or main with your own call to [RequestServer.trySimulatedRequest], you can simulate requests from your command-ine shell. Call the program like this: 194 195 $(CONSOLE 196 ./yourprogram GET / name=adr 197 ) 198 199 And it will print the result to stdout instead of running a server, regardless of build more.. 200 201 CGI_Setup_tips: 202 203 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file to set a particular file to be run through the cgi program. Note that all "subdirectories" of it also run the program; if you configure `/foo` to be a cgi script, then going to `/foo/bar` will call your cgi handler function with `cgi.pathInfo == "/bar"`. 204 205 Overview_Of_Basic_Concepts: 206 207 cgi.d offers both lower-level handler apis as well as higher-level auto-dispatcher apis. For a lower-level handler function, you'll probably want to review the following functions: 208 209 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 210 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 211 212 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 213 214 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 215 216 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 217 218 Redirections: [Cgi.setResponseLocation] 219 220 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 221 222 Websockets: [Websocket], [websocketRequested], [acceptWebsocket]. For websockets, use the `embedded_httpd_hybrid` build mode for best results, because it is optimized for handling large numbers of idle connections compared to the other build modes. 223 224 Overriding behavior for special cases streaming input data: see the virtual functions [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 225 226 A basic program using the lower-level api might look like: 227 228 --- 229 import arsd.cgi; 230 231 // you write a request handler which always takes a Cgi object 232 void handler(Cgi cgi) { 233 /+ 234 when the user goes to your site, suppose you are being hosted at http://example.com/yourapp 235 236 If the user goes to http://example.com/yourapp/test?name=value 237 then the url will be parsed out into the following pieces: 238 239 cgi.pathInfo == "/test". This is everything after yourapp's name. (If you are doing an embedded http server, your app's name is blank, so pathInfo will be the whole path of the url.) 240 241 cgi.scriptName == "yourapp". With an embedded http server, this will be blank. 242 243 cgi.host == "example.com" 244 245 cgi.https == false 246 247 cgi.queryString == "name=value" (there's also cgi.search, which will be "?name=value", including the ?) 248 249 The query string is further parsed into the `get` and `getArray` members, so: 250 251 cgi.get == ["name": "value"], meaning you can do `cgi.get["name"] == "value"` 252 253 And 254 255 cgi.getArray == ["name": ["value"]]. 256 257 Why is there both `get` and `getArray`? The standard allows names to be repeated. This can be very useful, 258 it is how http forms naturally pass multiple items like a set of checkboxes. So `getArray` is the complete data 259 if you need it. But since so often you only care about one value, the `get` member provides more convenient access. 260 261 We can use these members to process the request and build link urls. Other info from the request are in other members, we'll look at them later. 262 +/ 263 switch(cgi.pathInfo) { 264 // the home page will be a small html form that can set a cookie. 265 case "/": 266 cgi.write(`<!DOCTYPE html> 267 <html> 268 <body> 269 <form method="POST" action="set-cookie"> 270 <label>Your name: <input type="text" name="name" /></label> 271 <input type="submit" value="Submit" /> 272 </form> 273 </body> 274 </html> 275 `, true); // the , true tells it that this is the one, complete response i want to send, allowing some optimizations. 276 break; 277 // POSTing to this will set a cookie with our submitted name 278 case "/set-cookie": 279 // HTTP has a number of request methods (also called "verbs") to tell 280 // what you should do with the given resource. 281 // The most common are GET and POST, the ones used in html forms. 282 // You can check which one was used with the `cgi.requestMethod` property. 283 if(cgi.requestMethod == Cgi.RequestMethod.POST) { 284 285 // headers like redirections need to be set before we call `write` 286 cgi.setResponseLocation("read-cookie"); 287 288 // just like how url params go into cgi.get/getArray, form data submitted in a POST 289 // body go to cgi.post/postArray. Please note that a POST request can also have get 290 // params in addition to post params. 291 // 292 // There's also a convenience function `cgi.request("name")` which checks post first, 293 // then get if it isn't found there, and then returns a default value if it is in neither. 294 if("name" in cgi.post) { 295 // we can set cookies with a method too 296 // again, cookies need to be set before calling `cgi.write`, since they 297 // are a kind of header. 298 cgi.setCookie("name" , cgi.post["name"]); 299 } 300 301 // the user will probably never see this, since the response location 302 // is an automatic redirect, but it is still best to say something anyway 303 cgi.write("Redirecting you to see the cookie...", true); 304 } else { 305 // you can write out response codes and headers 306 // as well as response bodies 307 // 308 // But always check the cgi docs before using the generic 309 // `header` method - if there is a specific method for your 310 // header, use it before resorting to the generic one to avoid 311 // a header value from being sent twice. 312 cgi.setResponseLocation("405 Method Not Allowed"); 313 // there is no special accept member, so you can use the generic header function 314 cgi.header("Accept: POST"); 315 // but content type does have a method, so prefer to use it: 316 cgi.setResponseContentType("text/plain"); 317 318 // all the headers are buffered, and will be sent upon the first body 319 // write. you can actually modify some of them before sending if need be. 320 cgi.write("You must use the POST http verb on this resource.", true); 321 } 322 break; 323 // and GETting this will read the cookie back out 324 case "/read-cookie": 325 // I did NOT pass `,true` here because this is writing a partial response. 326 // It is possible to stream data to the user in chunks by writing partial 327 // responses the calling `cgi.flush();` to send the partial response immediately. 328 // normally, you'd only send partial chunks if you have to - it is better to build 329 // a response as a whole and send it as a whole whenever possible - but here I want 330 // to demo that you can. 331 cgi.write("Hello, "); 332 if("name" in cgi.cookies) { 333 import arsd.dom; // dom.d provides a lot of helpers for html 334 // since the cookie is set, we need to write it out properly to 335 // avoid cross-site scripting attacks. 336 // 337 // Getting this stuff right automatically is a benefit of using the higher 338 // level apis, but this demo is to show the fundamental building blocks, so 339 // we're responsible to take care of it. 340 cgi.write(htmlEntitiesEncode(cgi.cookies["name"])); 341 } else { 342 cgi.write("friend"); 343 } 344 345 // note that I never called cgi.setResponseContentType, since the default is text/html. 346 // it doesn't hurt to do it explicitly though, just remember to do it before any cgi.write 347 // calls. 348 break; 349 default: 350 // no path matched 351 cgi.setResponseStatus("404 Not Found"); 352 cgi.write("Resource not found.", true); 353 } 354 } 355 356 // and this adds the boilerplate to set up a server according to the 357 // compile version configuration and call your handler as requests come in 358 mixin GenericMain!handler; // the `handler` here is the name of your function 359 --- 360 361 Even if you plan to always use the higher-level apis, I still recommend you at least familiarize yourself with the lower level functions, since they provide the lightest weight, most flexible options to get down to business if you ever need them. 362 363 In the lower-level api, the [Cgi] object represents your HTTP transaction. It has functions to describe the request and for you to send your response. It leaves the details of how you o it up to you. The general guideline though is to avoid depending any variables outside your handler function, since there's no guarantee they will survive to another handler. You can use global vars as a lazy initialized cache, but you should always be ready in case it is empty. (One exception: if you use `-version=embedded_httpd_threads -version=cgi_no_fork`, then you can rely on it more, but you should still really write things assuming your function won't have anything survive beyond its return for max scalability and compatibility.) 364 365 A basic program using the higher-level apis might look like: 366 367 --- 368 /+ 369 import arsd.cgi; 370 371 struct LoginData { 372 string currentUser; 373 } 374 375 class AppClass : WebObject { 376 string foo() {} 377 } 378 379 mixin DispatcherMain!( 380 "/assets/.serveStaticFileDirectory("assets/", true), // serve the files in the assets subdirectory 381 "/".serveApi!AppClass, 382 "/thing/".serveRestObject, 383 ); 384 +/ 385 --- 386 387 Guide_for_PHP_users: 388 (Please note: I wrote this section in 2008. A lot of PHP hosts still ran 4.x back then, so it was common to avoid using classes - introduced in php 5 - to maintain compatibility! If you're coming from php more recently, this may not be relevant anymore, but still might help you.) 389 390 If you are coming from old-style PHP, here's a quick guide to help you get started: 391 392 $(SIDE_BY_SIDE 393 $(COLUMN 394 ```php 395 <?php 396 $foo = $_POST["foo"]; 397 $bar = $_GET["bar"]; 398 $baz = $_COOKIE["baz"]; 399 400 $user_ip = $_SERVER["REMOTE_ADDR"]; 401 $host = $_SERVER["HTTP_HOST"]; 402 $path = $_SERVER["PATH_INFO"]; 403 404 setcookie("baz", "some value"); 405 406 echo "hello!"; 407 ?> 408 ``` 409 ) 410 $(COLUMN 411 --- 412 import arsd.cgi; 413 void app(Cgi cgi) { 414 string foo = cgi.post["foo"]; 415 string bar = cgi.get["bar"]; 416 string baz = cgi.cookies["baz"]; 417 418 string user_ip = cgi.remoteAddress; 419 string host = cgi.host; 420 string path = cgi.pathInfo; 421 422 cgi.setCookie("baz", "some value"); 423 424 cgi.write("hello!"); 425 } 426 427 mixin GenericMain!app 428 --- 429 ) 430 ) 431 432 $(H3 Array elements) 433 434 435 In PHP, you can give a form element a name like `"something[]"`, and then 436 `$_POST["something"]` gives an array. In D, you can use whatever name 437 you want, and access an array of values with the `cgi.getArray["name"]` and 438 `cgi.postArray["name"]` members. 439 440 $(H3 Databases) 441 442 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 443 of these, but the rest of my arsd repository has much of it. For example, 444 to access a MySQL database, download `database.d` and `mysql.d` from my 445 github repo, and try this code (assuming, of course, your database is 446 set up): 447 448 --- 449 import arsd.cgi; 450 import arsd.mysql; 451 452 void app(Cgi cgi) { 453 auto database = new MySql("localhost", "username", "password", "database_name"); 454 foreach(row; mysql.query("SELECT count(id) FROM people")) 455 cgi.write(row[0] ~ " people in database"); 456 } 457 458 mixin GenericMain!app; 459 --- 460 461 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 462 implementing the same basic interface. 463 464 See_Also: 465 466 You may also want to see [arsd.dom], [arsd.webtemplate], and maybe some functions from my old [arsd.html] for more code for making 467 web applications. dom and webtemplate are used by the higher-level api here in cgi.d. 468 469 For working with json, try [arsd.jsvar]. 470 471 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 472 accessing databases. 473 474 If you are looking to access a web application via HTTP, try [arsd.http2]. 475 476 Copyright: 477 478 cgi.d copyright 2008-2023, Adam D. Ruppe. Provided under the Boost Software License. 479 480 Yes, this file is old, and yes, it is still actively maintained and used. 481 482 History: 483 An import of `arsd.core` was added on March 21, 2023 (dub v11.0). Prior to this, the module's default configuration was completely stand-alone. You must now include the `core.d` file in your builds with `cgi.d`. 484 485 This change is primarily to integrate the event loops across the library, allowing you to more easily use cgi.d along with my other libraries like simpledisplay and http2.d. Previously, you'd have to run separate helper threads. Now, they can all automatically work together. 486 +/ 487 module arsd.cgi; 488 489 static import arsd.core; 490 version(Posix) 491 import arsd.core : makeNonBlocking; 492 493 // FIXME: Nullable!T can be a checkbox that enables/disables the T on the automatic form 494 // and a SumType!(T, R) can be a radio box to pick between T and R to disclose the extra boxes on the automatic form 495 496 /++ 497 This micro-example uses the [dispatcher] api to act as a simple http file server, serving files found in the current directory and its children. 498 +/ 499 version(Demo) 500 unittest { 501 import arsd.cgi; 502 503 mixin DispatcherMain!( 504 "/".serveStaticFileDirectory(null, true) 505 ); 506 } 507 508 /++ 509 Same as the previous example, but written out long-form without the use of [DispatcherMain] nor [GenericMain]. 510 +/ 511 version(Demo) 512 unittest { 513 import arsd.cgi; 514 515 void requestHandler(Cgi cgi) { 516 cgi.dispatcher!( 517 "/".serveStaticFileDirectory(null, true) 518 ); 519 } 520 521 // mixin GenericMain!requestHandler would add this function: 522 void main(string[] args) { 523 // this is all the content of [cgiMainImpl] which you can also call 524 525 // cgi.d embeds a few add on functions like real time event forwarders 526 // and session servers it can run in other processes. this spawns them, if needed. 527 if(tryAddonServers(args)) 528 return; 529 530 // cgi.d allows you to easily simulate http requests from the command line, 531 // without actually starting a server. this function will do that. 532 if(trySimulatedRequest!(requestHandler, Cgi)(args)) 533 return; 534 535 RequestServer server; 536 // you can change the default port here if you like 537 // server.listeningPort = 9000; 538 539 // then call this to let the command line args override your default 540 server.configureFromCommandLine(args); 541 542 // here is where you could print out the listeningPort to the user if you wanted 543 544 // and serve the request(s) according to the compile configuration 545 server.serve!(requestHandler)(); 546 547 // or you could explicitly choose a serve mode like this: 548 // server.serveEmbeddedHttp!requestHandler(); 549 } 550 } 551 552 /++ 553 cgi.d has built-in testing helpers too. These will provide mock requests and mock sessions that 554 otherwise run through the rest of the internal mechanisms to call your functions without actually 555 spinning up a server. 556 +/ 557 version(Demo) 558 unittest { 559 import arsd.cgi; 560 561 void requestHandler(Cgi cgi) { 562 563 } 564 565 // D doesn't let me embed a unittest inside an example unittest 566 // so this is a function, but you can do it however in your real program 567 /* unittest */ void runTests() { 568 auto tester = new CgiTester(&requestHandler); 569 570 auto response = tester.GET("/"); 571 assert(response.code == 200); 572 } 573 } 574 575 static import std.file; 576 577 // for a single thread, linear request thing, use: 578 // -version=embedded_httpd_threads -version=cgi_no_threads 579 580 version(Posix) { 581 version(CRuntime_Musl) { 582 583 } else version(minimal) { 584 585 } else { 586 version(FreeBSD) { 587 // I never implemented the fancy stuff there either 588 } else { 589 version=with_breaking_cgi_features; 590 version=with_sendfd; 591 version=with_addon_servers; 592 } 593 } 594 } 595 596 version(Windows) { 597 version(minimal) { 598 599 } else { 600 // not too concerned about gdc here since the mingw version is fairly new as well 601 version=with_breaking_cgi_features; 602 } 603 } 604 605 // FIXME: can use the arsd.core function now but it is trivial anyway tbh 606 void cloexec(int fd) { 607 version(Posix) { 608 import core.sys.posix.fcntl; 609 fcntl(fd, F_SETFD, FD_CLOEXEC); 610 } 611 } 612 613 void cloexec(Socket s) { 614 version(Posix) { 615 import core.sys.posix.fcntl; 616 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 617 } 618 } 619 620 // the servers must know about the connections to talk to them; the interfaces are vital 621 version(with_addon_servers) 622 version=with_addon_servers_connections; 623 624 version(embedded_httpd) { 625 version=embedded_httpd_hybrid; 626 /* 627 version(with_openssl) { 628 pragma(lib, "crypto"); 629 pragma(lib, "ssl"); 630 } 631 */ 632 } 633 634 version(embedded_httpd_hybrid) { 635 version=embedded_httpd_threads; 636 version(cgi_no_fork) {} else version(Posix) 637 version=cgi_use_fork; 638 version=cgi_use_fiber; 639 } 640 641 version(cgi_use_fork) 642 enum cgi_use_fork_default = true; 643 else 644 enum cgi_use_fork_default = false; 645 646 version(embedded_httpd_processes) 647 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 648 649 version(embedded_httpd_threads) { 650 // unless the user overrides the default.. 651 version(cgi_session_server_process) 652 {} 653 else 654 version=cgi_embedded_sessions; 655 } 656 version(scgi) { 657 // unless the user overrides the default.. 658 version(cgi_session_server_process) 659 {} 660 else 661 version=cgi_embedded_sessions; 662 } 663 664 // fall back if the other is not defined so we can cleanly version it below 665 version(cgi_embedded_sessions) {} 666 else version=cgi_session_server_process; 667 668 669 version=cgi_with_websocket; 670 671 enum long defaultMaxContentLength = 5_000_000; 672 673 /* 674 675 To do a file download offer in the browser: 676 677 cgi.setResponseContentType("text/csv"); 678 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 679 */ 680 681 // FIXME: the location header is supposed to be an absolute url I guess. 682 683 // FIXME: would be cool to flush part of a dom document before complete 684 // somehow in here and dom.d. 685 686 687 // these are public so you can mixin GenericMain. 688 // FIXME: use a function level import instead! 689 public import std.string; 690 public import std.stdio; 691 public import std.conv; 692 import std.uri; 693 import std.uni; 694 import std.algorithm.comparison; 695 import std.algorithm.searching; 696 import std.exception; 697 import std.base64; 698 static import std.algorithm; 699 import std.datetime; 700 import std.range; 701 702 import std.process; 703 704 import std.zlib; 705 706 707 T[] consume(T)(T[] range, int count) { 708 if(count > range.length) 709 count = range.length; 710 return range[count..$]; 711 } 712 713 int locationOf(T)(T[] data, string item) { 714 const(ubyte[]) d = cast(const(ubyte[])) data; 715 const(ubyte[]) i = cast(const(ubyte[])) item; 716 717 // this is a vague sanity check to ensure we aren't getting insanely 718 // sized input that will infinite loop below. it should never happen; 719 // even huge file uploads ought to come in smaller individual pieces. 720 if(d.length > (int.max/2)) 721 throw new Exception("excessive block of input"); 722 723 for(int a = 0; a < d.length; a++) { 724 if(a + i.length > d.length) 725 return -1; 726 if(d[a..a+i.length] == i) 727 return a; 728 } 729 730 return -1; 731 } 732 733 /// If you are doing a custom cgi class, mixing this in can take care of 734 /// the required constructors for you 735 mixin template ForwardCgiConstructors() { 736 this(long maxContentLength = defaultMaxContentLength, 737 string[string] env = null, 738 const(ubyte)[] delegate() readdata = null, 739 void delegate(const(ubyte)[]) _rawDataOutput = null, 740 void delegate() _flush = null 741 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 742 743 this(string[] args) { super(args); } 744 745 this( 746 BufferedInputRange inputData, 747 string address, ushort _port, 748 int pathInfoStarts = 0, 749 bool _https = false, 750 void delegate(const(ubyte)[]) _rawDataOutput = null, 751 void delegate() _flush = null, 752 // this pointer tells if the connection is supposed to be closed after we handle this 753 bool* closeConnection = null) 754 { 755 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 756 } 757 758 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 759 } 760 761 /// thrown when a connection is closed remotely while we waiting on data from it 762 class ConnectionClosedException : Exception { 763 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 764 super(message, file, line, next); 765 } 766 } 767 768 769 version(Windows) { 770 // FIXME: ugly hack to solve stdin exception problems on Windows: 771 // reading stdin results in StdioException (Bad file descriptor) 772 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 773 private struct stdin { 774 struct ByChunk { // Replicates std.stdio.ByChunk 775 private: 776 ubyte[] chunk_; 777 public: 778 this(size_t size) 779 in { 780 assert(size, "size must be larger than 0"); 781 } 782 do { 783 chunk_ = new ubyte[](size); 784 popFront(); 785 } 786 787 @property bool empty() const { 788 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 789 } 790 @property nothrow ubyte[] front() { return chunk_; } 791 void popFront() { 792 enforce(!empty, "Cannot call popFront on empty range"); 793 chunk_ = stdin.rawRead(chunk_); 794 } 795 } 796 797 import core.sys.windows.windows; 798 static: 799 800 T[] rawRead(T)(T[] buf) { 801 uint bytesRead; 802 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 803 804 if (!result) { 805 auto err = GetLastError(); 806 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 807 return buf[0..0]; 808 // Some other error, throw it 809 810 char* buffer; 811 scope(exit) LocalFree(buffer); 812 813 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 814 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 815 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 816 throw new Exception(to!string(buffer)); 817 } 818 enforce(!(bytesRead % T.sizeof), "I/O error"); 819 return buf[0..bytesRead / T.sizeof]; 820 } 821 822 auto byChunk(size_t sz) { return ByChunk(sz); } 823 824 void close() { 825 std.stdio.stdin.close; 826 } 827 } 828 } 829 830 /// The main interface with the web request 831 class Cgi { 832 public: 833 /// the methods a request can be 834 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 835 // these are defined in the standard, but idk if they are useful for anything 836 OPTIONS, TRACE, CONNECT, 837 // These seem new, I have only recently seen them 838 PATCH, MERGE, 839 // this is an extension for when the method is not specified and you want to assume 840 CommandLine } 841 842 843 /+ 844 /++ 845 Cgi provides a per-request memory pool 846 847 +/ 848 void[] allocateMemory(size_t nBytes) { 849 850 } 851 852 /// ditto 853 void[] reallocateMemory(void[] old, size_t nBytes) { 854 855 } 856 857 /// ditto 858 void freeMemory(void[] memory) { 859 860 } 861 +/ 862 863 864 /* 865 import core.runtime; 866 auto args = Runtime.args(); 867 868 we can call the app a few ways: 869 870 1) set up the environment variables and call the app (manually simulating CGI) 871 2) simulate a call automatically: 872 ./app method 'uri' 873 874 for example: 875 ./app get /path?arg arg2=something 876 877 Anything on the uri is treated as query string etc 878 879 on get method, further args are appended to the query string (encoded automatically) 880 on post method, further args are done as post 881 882 883 @name means import from file "name". if name == -, it uses stdin 884 (so info=@- means set info to the value of stdin) 885 886 887 Other arguments include: 888 --cookie name=value (these are all concated together) 889 --header 'X-Something: cool' 890 --referrer 'something' 891 --port 80 892 --remote-address some.ip.address.here 893 --https yes 894 --user-agent 'something' 895 --userpass 'user:pass' 896 --authorization 'Basic base64encoded_user:pass' 897 --accept 'content' // FIXME: better example 898 --last-event-id 'something' 899 --host 'something.com' 900 901 Non-simulation arguments: 902 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 903 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 904 905 */ 906 907 /** Initializes it with command line arguments (for easy testing) */ 908 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 909 rawDataOutput = _rawDataOutput; 910 // these are all set locally so the loop works 911 // without triggering errors in dmd 2.064 912 // we go ahead and set them at the end of it to the this version 913 int port; 914 string referrer; 915 string remoteAddress; 916 string userAgent; 917 string authorization; 918 string origin; 919 string accept; 920 string lastEventId; 921 bool https; 922 string host; 923 RequestMethod requestMethod; 924 string requestUri; 925 string pathInfo; 926 string queryString; 927 928 bool lookingForMethod; 929 bool lookingForUri; 930 string nextArgIs; 931 932 string _cookie; 933 string _queryString; 934 string[][string] _post; 935 string[string] _headers; 936 937 string[] breakUp(string s) { 938 string k, v; 939 auto idx = s.indexOf("="); 940 if(idx == -1) { 941 k = s; 942 } else { 943 k = s[0 .. idx]; 944 v = s[idx + 1 .. $]; 945 } 946 947 return [k, v]; 948 } 949 950 lookingForMethod = true; 951 952 scriptName = args[0]; 953 scriptFileName = args[0]; 954 955 environmentVariables = cast(const) environment.toAA; 956 957 foreach(arg; args[1 .. $]) { 958 if(arg.startsWith("--")) { 959 nextArgIs = arg[2 .. $]; 960 } else if(nextArgIs.length) { 961 if (nextArgIs == "cookie") { 962 auto info = breakUp(arg); 963 if(_cookie.length) 964 _cookie ~= "; "; 965 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 966 } 967 else if (nextArgIs == "port") { 968 port = to!int(arg); 969 } 970 else if (nextArgIs == "referrer") { 971 referrer = arg; 972 } 973 else if (nextArgIs == "remote-address") { 974 remoteAddress = arg; 975 } 976 else if (nextArgIs == "user-agent") { 977 userAgent = arg; 978 } 979 else if (nextArgIs == "authorization") { 980 authorization = arg; 981 } 982 else if (nextArgIs == "userpass") { 983 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 984 } 985 else if (nextArgIs == "origin") { 986 origin = arg; 987 } 988 else if (nextArgIs == "accept") { 989 accept = arg; 990 } 991 else if (nextArgIs == "last-event-id") { 992 lastEventId = arg; 993 } 994 else if (nextArgIs == "https") { 995 if(arg == "yes") 996 https = true; 997 } 998 else if (nextArgIs == "header") { 999 string thing, other; 1000 auto idx = arg.indexOf(":"); 1001 if(idx == -1) 1002 throw new Exception("need a colon in a http header"); 1003 thing = arg[0 .. idx]; 1004 other = arg[idx + 1.. $]; 1005 _headers[thing.strip.toLower()] = other.strip; 1006 } 1007 else if (nextArgIs == "host") { 1008 host = arg; 1009 } 1010 // else 1011 // skip, we don't know it but that's ok, it might be used elsewhere so no error 1012 1013 nextArgIs = null; 1014 } else if(lookingForMethod) { 1015 lookingForMethod = false; 1016 lookingForUri = true; 1017 1018 if(arg.asLowerCase().equal("commandline")) 1019 requestMethod = RequestMethod.CommandLine; 1020 else 1021 requestMethod = to!RequestMethod(arg.toUpper()); 1022 } else if(lookingForUri) { 1023 lookingForUri = false; 1024 1025 requestUri = arg; 1026 1027 auto idx = arg.indexOf("?"); 1028 if(idx == -1) 1029 pathInfo = arg; 1030 else { 1031 pathInfo = arg[0 .. idx]; 1032 _queryString = arg[idx + 1 .. $]; 1033 } 1034 } else { 1035 // it is an argument of some sort 1036 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1037 auto parts = breakUp(arg); 1038 _post[parts[0]] ~= parts[1]; 1039 allPostNamesInOrder ~= parts[0]; 1040 allPostValuesInOrder ~= parts[1]; 1041 } else { 1042 if(_queryString.length) 1043 _queryString ~= "&"; 1044 auto parts = breakUp(arg); 1045 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 1046 } 1047 } 1048 } 1049 1050 acceptsGzip = false; 1051 keepAliveRequested = false; 1052 requestHeaders = cast(immutable) _headers; 1053 1054 cookie = _cookie; 1055 cookiesArray = getCookieArray(); 1056 cookies = keepLastOf(cookiesArray); 1057 1058 queryString = _queryString; 1059 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1060 get = keepLastOf(getArray); 1061 1062 postArray = cast(immutable) _post; 1063 post = keepLastOf(_post); 1064 1065 // FIXME 1066 filesArray = null; 1067 files = null; 1068 1069 isCalledWithCommandLineArguments = true; 1070 1071 this.port = port; 1072 this.referrer = referrer; 1073 this.remoteAddress = remoteAddress; 1074 this.userAgent = userAgent; 1075 this.authorization = authorization; 1076 this.origin = origin; 1077 this.accept = accept; 1078 this.lastEventId = lastEventId; 1079 this.https = https; 1080 this.host = host; 1081 this.requestMethod = requestMethod; 1082 this.requestUri = requestUri; 1083 this.pathInfo = pathInfo; 1084 this.queryString = queryString; 1085 this.postBody = null; 1086 } 1087 1088 private { 1089 string[] allPostNamesInOrder; 1090 string[] allPostValuesInOrder; 1091 string[] allGetNamesInOrder; 1092 string[] allGetValuesInOrder; 1093 } 1094 1095 CgiConnectionHandle getOutputFileHandle() { 1096 return _outputFileHandle; 1097 } 1098 1099 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 1100 1101 /** Initializes it using a CGI or CGI-like interface */ 1102 this(long maxContentLength = defaultMaxContentLength, 1103 // use this to override the environment variable listing 1104 in string[string] env = null, 1105 // and this should return a chunk of data. return empty when done 1106 const(ubyte)[] delegate() readdata = null, 1107 // finally, use this to do custom output if needed 1108 void delegate(const(ubyte)[]) _rawDataOutput = null, 1109 // to flush teh custom output 1110 void delegate() _flush = null 1111 ) 1112 { 1113 1114 // these are all set locally so the loop works 1115 // without triggering errors in dmd 2.064 1116 // we go ahead and set them at the end of it to the this version 1117 int port; 1118 string referrer; 1119 string remoteAddress; 1120 string userAgent; 1121 string authorization; 1122 string origin; 1123 string accept; 1124 string lastEventId; 1125 bool https; 1126 string host; 1127 RequestMethod requestMethod; 1128 string requestUri; 1129 string pathInfo; 1130 string queryString; 1131 1132 1133 1134 isCalledWithCommandLineArguments = false; 1135 rawDataOutput = _rawDataOutput; 1136 flushDelegate = _flush; 1137 auto getenv = delegate string(string var) { 1138 if(env is null) 1139 return std.process.environment.get(var); 1140 auto e = var in env; 1141 if(e is null) 1142 return null; 1143 return *e; 1144 }; 1145 1146 environmentVariables = env is null ? 1147 cast(const) environment.toAA : 1148 env; 1149 1150 // fetching all the request headers 1151 string[string] requestHeadersHere; 1152 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 1153 if(k.startsWith("HTTP_")) { 1154 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 1155 } 1156 } 1157 1158 this.requestHeaders = assumeUnique(requestHeadersHere); 1159 1160 requestUri = getenv("REQUEST_URI"); 1161 1162 cookie = getenv("HTTP_COOKIE"); 1163 cookiesArray = getCookieArray(); 1164 cookies = keepLastOf(cookiesArray); 1165 1166 referrer = getenv("HTTP_REFERER"); 1167 userAgent = getenv("HTTP_USER_AGENT"); 1168 remoteAddress = getenv("REMOTE_ADDR"); 1169 host = getenv("HTTP_HOST"); 1170 pathInfo = getenv("PATH_INFO"); 1171 1172 queryString = getenv("QUERY_STRING"); 1173 scriptName = getenv("SCRIPT_NAME"); 1174 { 1175 import core.runtime; 1176 auto sfn = getenv("SCRIPT_FILENAME"); 1177 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 1178 } 1179 1180 bool iis = false; 1181 1182 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 1183 if(requestUri.length == 0) { 1184 // IIS sometimes includes the script name as part of the path info - we don't want that 1185 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 1186 pathInfo = pathInfo[scriptName.length .. $]; 1187 1188 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 1189 1190 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 1191 1192 // FIXME: this works for apache and iis... but what about others? 1193 } 1194 1195 1196 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1197 getArray = assumeUnique(ugh); 1198 get = keepLastOf(getArray); 1199 1200 1201 // NOTE: on shitpache, you need to specifically forward this 1202 authorization = getenv("HTTP_AUTHORIZATION"); 1203 // this is a hack because Apache is a shitload of fuck and 1204 // refuses to send the real header to us. Compatible 1205 // programs should send both the standard and X- versions 1206 1207 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 1208 // unnecessary with mod_rewrite, so it is commented 1209 1210 //if(authorization.length == 0) // if the std is there, use it 1211 // authorization = getenv("HTTP_X_AUTHORIZATION"); 1212 1213 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 1214 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 1215 port = to!int(getenv("SERVER_PORT")); 1216 else 1217 port = 0; // this was probably called from the command line 1218 1219 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1220 if(ae.length && ae.indexOf("gzip") != -1) 1221 acceptsGzip = true; 1222 1223 accept = getenv("HTTP_ACCEPT"); 1224 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1225 1226 auto ka = getenv("HTTP_CONNECTION"); 1227 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1228 keepAliveRequested = true; 1229 1230 auto or = getenv("HTTP_ORIGIN"); 1231 origin = or; 1232 1233 auto rm = getenv("REQUEST_METHOD"); 1234 if(rm.length) 1235 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1236 else 1237 requestMethod = RequestMethod.CommandLine; 1238 1239 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1240 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1241 1242 // FIXME: DOCUMENT_ROOT? 1243 1244 // FIXME: what about PUT? 1245 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1246 version(preserveData) // a hack to make forwarding simpler 1247 immutable(ubyte)[] data; 1248 size_t amountReceived = 0; 1249 auto contentType = getenv("CONTENT_TYPE"); 1250 1251 // FIXME: is this ever not going to be set? I guess it depends 1252 // on if the server de-chunks and buffers... seems like it has potential 1253 // to be slow if they did that. The spec says it is always there though. 1254 // And it has worked reliably for me all year in the live environment, 1255 // but some servers might be different. 1256 auto cls = getenv("CONTENT_LENGTH"); 1257 auto contentLength = to!size_t(cls.length ? cls : "0"); 1258 1259 immutable originalContentLength = contentLength; 1260 if(contentLength) { 1261 if(maxContentLength > 0 && contentLength > maxContentLength) { 1262 setResponseStatus("413 Request entity too large"); 1263 write("You tried to upload a file that is too large."); 1264 close(); 1265 throw new Exception("POST too large"); 1266 } 1267 prepareForIncomingDataChunks(contentType, contentLength); 1268 1269 1270 int processChunk(in ubyte[] chunk) { 1271 if(chunk.length > contentLength) { 1272 handleIncomingDataChunk(chunk[0..contentLength]); 1273 amountReceived += contentLength; 1274 contentLength = 0; 1275 return 1; 1276 } else { 1277 handleIncomingDataChunk(chunk); 1278 contentLength -= chunk.length; 1279 amountReceived += chunk.length; 1280 } 1281 if(contentLength == 0) 1282 return 1; 1283 1284 onRequestBodyDataReceived(amountReceived, originalContentLength); 1285 return 0; 1286 } 1287 1288 1289 if(readdata is null) { 1290 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1291 if(processChunk(chunk)) 1292 break; 1293 } else { 1294 // we have a custom data source.. 1295 auto chunk = readdata(); 1296 while(chunk.length) { 1297 if(processChunk(chunk)) 1298 break; 1299 chunk = readdata(); 1300 } 1301 } 1302 1303 onRequestBodyDataReceived(amountReceived, originalContentLength); 1304 postArray = assumeUnique(pps._post); 1305 filesArray = assumeUnique(pps._files); 1306 files = keepLastOf(filesArray); 1307 post = keepLastOf(postArray); 1308 this.postBody = pps.postBody; 1309 cleanUpPostDataState(); 1310 } 1311 1312 version(preserveData) 1313 originalPostData = data; 1314 } 1315 // fixme: remote_user script name 1316 1317 1318 this.port = port; 1319 this.referrer = referrer; 1320 this.remoteAddress = remoteAddress; 1321 this.userAgent = userAgent; 1322 this.authorization = authorization; 1323 this.origin = origin; 1324 this.accept = accept; 1325 this.lastEventId = lastEventId; 1326 this.https = https; 1327 this.host = host; 1328 this.requestMethod = requestMethod; 1329 this.requestUri = requestUri; 1330 this.pathInfo = pathInfo; 1331 this.queryString = queryString; 1332 } 1333 1334 /// Cleans up any temporary files. Do not use the object 1335 /// after calling this. 1336 /// 1337 /// NOTE: it is called automatically by GenericMain 1338 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1339 void dispose() { 1340 foreach(file; files) { 1341 if(!file.contentInMemory) 1342 if(std.file.exists(file.contentFilename)) 1343 std.file.remove(file.contentFilename); 1344 } 1345 } 1346 1347 private { 1348 struct PostParserState { 1349 string contentType; 1350 string boundary; 1351 string localBoundary; // the ones used at the end or something lol 1352 bool isMultipart; 1353 bool needsSavedBody; 1354 1355 ulong expectedLength; 1356 ulong contentConsumed; 1357 immutable(ubyte)[] buffer; 1358 1359 // multipart parsing state 1360 int whatDoWeWant; 1361 bool weHaveAPart; 1362 string[] thisOnesHeaders; 1363 immutable(ubyte)[] thisOnesData; 1364 1365 string postBody; 1366 1367 UploadedFile piece; 1368 bool isFile = false; 1369 1370 size_t memoryCommitted; 1371 1372 // do NOT keep mutable references to these anywhere! 1373 // I assume they are unique in the constructor once we're all done getting data. 1374 string[][string] _post; 1375 UploadedFile[][string] _files; 1376 } 1377 1378 PostParserState pps; 1379 } 1380 1381 /// This represents a file the user uploaded via a POST request. 1382 static struct UploadedFile { 1383 /// If you want to create one of these structs for yourself from some data, 1384 /// use this function. 1385 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1386 Cgi.UploadedFile f; 1387 f.filename = name; 1388 f.content = cast(immutable(ubyte)[]) data; 1389 f.contentInMemory = true; 1390 return f; 1391 } 1392 1393 string name; /// The name of the form element. 1394 string filename; /// The filename the user set. 1395 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1396 1397 /** 1398 For small files, cgi.d will buffer the uploaded file in memory, and make it 1399 directly accessible to you through the content member. I find this very convenient 1400 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1401 often want to inspect and modify the file anyway!) 1402 1403 I find the file is very large, it is undesirable to eat that much memory just 1404 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1405 to the constructor so they are accepted, cgi.d will write the content to a temporary 1406 file that you can re-read later. 1407 1408 You can override this behavior by subclassing Cgi and overriding the protected 1409 handlePostChunk method. Note that the object is not initialized when you 1410 write that method - the http headers are available, but the cgi.post method 1411 is not. You may parse the file as it streams in using this method. 1412 1413 1414 Anyway, if the file is small enough to be in memory, contentInMemory will be 1415 set to true, and the content is available in the content member. 1416 1417 If not, contentInMemory will be set to false, and the content saved in a file, 1418 whose name will be available in the contentFilename member. 1419 1420 1421 Tip: if you know you are always dealing with small files, and want the convenience 1422 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1423 a large file comes in, it simply throws an exception (and HTTP error response) 1424 instead of trying to handle it. 1425 1426 The default value of maxContentLength in the constructor is for small files. 1427 */ 1428 bool contentInMemory = true; // the default ought to always be true 1429 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1430 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1431 1432 /// 1433 ulong fileSize() const { 1434 if(contentInMemory) 1435 return content.length; 1436 import std.file; 1437 return std.file.getSize(contentFilename); 1438 1439 } 1440 1441 /// 1442 void writeToFile(string filenameToSaveTo) const { 1443 import std.file; 1444 if(contentInMemory) 1445 std.file.write(filenameToSaveTo, content); 1446 else 1447 std.file.rename(contentFilename, filenameToSaveTo); 1448 } 1449 } 1450 1451 // given a content type and length, decide what we're going to do with the data.. 1452 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1453 pps.expectedLength = contentLength; 1454 1455 auto terminator = contentType.indexOf(";"); 1456 if(terminator == -1) 1457 terminator = contentType.length; 1458 1459 pps.contentType = contentType[0 .. terminator]; 1460 auto b = contentType[terminator .. $]; 1461 if(b.length) { 1462 auto idx = b.indexOf("boundary="); 1463 if(idx != -1) { 1464 pps.boundary = b[idx + "boundary=".length .. $]; 1465 pps.localBoundary = "\r\n--" ~ pps.boundary; 1466 } 1467 } 1468 1469 // while a content type SHOULD be sent according to the RFC, it is 1470 // not required. We're told we SHOULD guess by looking at the content 1471 // but it seems to me that this only happens when it is urlencoded. 1472 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1473 pps.isMultipart = false; 1474 pps.needsSavedBody = false; 1475 } else if(pps.contentType == "multipart/form-data") { 1476 pps.isMultipart = true; 1477 enforce(pps.boundary.length, "no boundary"); 1478 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1479 // save the body so the application can handle it 1480 pps.isMultipart = false; 1481 pps.needsSavedBody = true; 1482 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1483 // save the body so the application can handle it 1484 pps.needsSavedBody = true; 1485 pps.isMultipart = false; 1486 } else { 1487 // the rest is 100% handled by the application. just save the body and send it to them 1488 pps.needsSavedBody = true; 1489 pps.isMultipart = false; 1490 } 1491 } 1492 1493 // handles streaming POST data. If you handle some other content type, you should 1494 // override this. If the data isn't the content type you want, you ought to call 1495 // super.handleIncomingDataChunk so regular forms and files still work. 1496 1497 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1498 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1499 // input anyway, so I'm not going to get too worked up about it right now. 1500 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1501 if(chunk.length == 0) 1502 return; 1503 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1504 // if we're passed big chunks, it might throw unnecessarily. 1505 // just pass it smaller chunks at a time. 1506 if(pps.isMultipart) { 1507 // multipart/form-data 1508 1509 1510 // FIXME: this might want to be factored out and factorized 1511 // need to make sure the stream hooks actually work. 1512 void pieceHasNewContent() { 1513 // we just grew the piece's buffer. Do we have to switch to file backing? 1514 if(pps.piece.contentInMemory) { 1515 if(pps.piece.content.length <= 10 * 1024 * 1024) 1516 // meh, I'm ok with it. 1517 return; 1518 else { 1519 // this is too big. 1520 if(!pps.isFile) 1521 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1522 else { 1523 // a file this large is probably acceptable though... let's use a backing file. 1524 pps.piece.contentInMemory = false; 1525 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1526 1527 int count = 0; 1528 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1529 // odds are this loop will never be entered, but we want it just in case. 1530 while(std.file.exists(pps.piece.contentFilename)) { 1531 count++; 1532 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1533 } 1534 // I hope this creates the file pretty quickly, or the loop might be useless... 1535 // FIXME: maybe I should write some kind of custom transaction here. 1536 std.file.write(pps.piece.contentFilename, pps.piece.content); 1537 1538 pps.piece.content = null; 1539 } 1540 } 1541 } else { 1542 // it's already in a file, so just append it to what we have 1543 if(pps.piece.content.length) { 1544 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1545 std.file.append(pps.piece.contentFilename, pps.piece.content); 1546 pps.piece.content = null; 1547 } 1548 } 1549 } 1550 1551 1552 void commitPart() { 1553 if(!pps.weHaveAPart) 1554 return; 1555 1556 pieceHasNewContent(); // be sure the new content is handled every time 1557 1558 if(pps.isFile) { 1559 // I'm not sure if other environments put files in post or not... 1560 // I used to not do it, but I think I should, since it is there... 1561 pps._post[pps.piece.name] ~= pps.piece.filename; 1562 pps._files[pps.piece.name] ~= pps.piece; 1563 1564 allPostNamesInOrder ~= pps.piece.name; 1565 allPostValuesInOrder ~= pps.piece.filename; 1566 } else { 1567 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1568 1569 allPostNamesInOrder ~= pps.piece.name; 1570 allPostValuesInOrder ~= cast(string) pps.piece.content; 1571 } 1572 1573 /* 1574 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1575 pps.piece.content.length < 1000 1576 ? 1577 to!string(pps.piece.content) 1578 : 1579 "too long"); 1580 */ 1581 1582 // FIXME: the limit here 1583 pps.memoryCommitted += pps.piece.content.length; 1584 1585 pps.weHaveAPart = false; 1586 pps.whatDoWeWant = 1; 1587 pps.thisOnesHeaders = null; 1588 pps.thisOnesData = null; 1589 1590 pps.piece = UploadedFile.init; 1591 pps.isFile = false; 1592 } 1593 1594 void acceptChunk() { 1595 pps.buffer ~= chunk; 1596 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1597 } 1598 1599 immutable(ubyte)[] consume(size_t howMuch) { 1600 pps.contentConsumed += howMuch; 1601 auto ret = pps.buffer[0 .. howMuch]; 1602 pps.buffer = pps.buffer[howMuch .. $]; 1603 return ret; 1604 } 1605 1606 dataConsumptionLoop: do { 1607 switch(pps.whatDoWeWant) { 1608 default: assert(0); 1609 case 0: 1610 acceptChunk(); 1611 // the format begins with two extra leading dashes, then we should be at the boundary 1612 if(pps.buffer.length < 2) 1613 return; 1614 assert(pps.buffer[0] == '-', "no leading dash"); 1615 consume(1); 1616 assert(pps.buffer[0] == '-', "no second leading dash"); 1617 consume(1); 1618 1619 pps.whatDoWeWant = 1; 1620 goto case 1; 1621 /* fallthrough */ 1622 case 1: // looking for headers 1623 // here, we should be lined up right at the boundary, which is followed by a \r\n 1624 1625 // want to keep the buffer under control in case we're under attack 1626 //stderr.writeln("here once"); 1627 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1628 // throw new Exception("wtf is up with the huge mime part headers"); 1629 1630 acceptChunk(); 1631 1632 if(pps.buffer.length < pps.boundary.length) 1633 return; // not enough data, since there should always be a boundary here at least 1634 1635 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1636 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1637 // we *should* be at the end here! 1638 assert(pps.buffer[0] == '-'); 1639 consume(1); 1640 assert(pps.buffer[0] == '-'); 1641 consume(1); 1642 1643 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1644 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1645 "not lined up on boundary " ~ pps.boundary); 1646 consume(pps.boundary.length); 1647 1648 assert(pps.buffer[0] == '-'); 1649 consume(1); 1650 assert(pps.buffer[0] == '-'); 1651 consume(1); 1652 1653 assert(pps.buffer[0] == '\r'); 1654 consume(1); 1655 assert(pps.buffer[0] == '\n'); 1656 consume(1); 1657 1658 assert(pps.buffer.length == 0); 1659 assert(pps.contentConsumed == pps.expectedLength); 1660 break dataConsumptionLoop; // we're done! 1661 } else { 1662 // we're not done yet. We should be lined up on a boundary. 1663 1664 // But, we want to ensure the headers are here before we consume anything! 1665 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1666 if(headerEndLocation == -1) 1667 return; // they *should* all be here, so we can handle them all at once. 1668 1669 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1670 "not lined up on boundary " ~ pps.boundary); 1671 1672 consume(pps.boundary.length); 1673 // the boundary is always followed by a \r\n 1674 assert(pps.buffer[0] == '\r'); 1675 consume(1); 1676 assert(pps.buffer[0] == '\n'); 1677 consume(1); 1678 } 1679 1680 // re-running since by consuming the boundary, we invalidate the old index. 1681 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1682 assert(headerEndLocation >= 0, "no header"); 1683 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1684 1685 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1686 1687 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1688 1689 // now we'll parse the headers 1690 foreach(h; pps.thisOnesHeaders) { 1691 auto p = h.indexOf(":"); 1692 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1693 string hn = h[0..p]; 1694 string hv = h[p+2..$]; 1695 1696 switch(hn.toLower) { 1697 default: assert(0); 1698 case "content-disposition": 1699 auto info = hv.split("; "); 1700 foreach(i; info[1..$]) { // skipping the form-data 1701 auto o = i.split("="); // FIXME 1702 string pn = o[0]; 1703 string pv = o[1][1..$-1]; 1704 1705 if(pn == "name") { 1706 pps.piece.name = pv; 1707 } else if (pn == "filename") { 1708 pps.piece.filename = pv; 1709 pps.isFile = true; 1710 } 1711 } 1712 break; 1713 case "content-type": 1714 pps.piece.contentType = hv; 1715 break; 1716 } 1717 } 1718 1719 pps.whatDoWeWant++; // move to the next step - the data 1720 break; 1721 case 2: 1722 // when we get here, pps.buffer should contain our first chunk of data 1723 1724 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1725 throw new Exception("wtf is up with the huge mime part buffer"); 1726 1727 acceptChunk(); 1728 1729 // so the trick is, we want to process all the data up to the boundary, 1730 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1731 // want to wait for the next chunk. We start by looking for the whole boundary 1732 // in the buffer somewhere. 1733 1734 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1735 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1736 if(boundaryLocation != -1) { 1737 // this is easy - we can see it in it's entirety! 1738 1739 pps.piece.content ~= consume(boundaryLocation); 1740 1741 assert(pps.buffer[0] == '\r'); 1742 consume(1); 1743 assert(pps.buffer[0] == '\n'); 1744 consume(1); 1745 assert(pps.buffer[0] == '-'); 1746 consume(1); 1747 assert(pps.buffer[0] == '-'); 1748 consume(1); 1749 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1750 pps.weHaveAPart = true; 1751 pps.whatDoWeWant = 1; // back to getting headers for the next part 1752 1753 commitPart(); // we're done here 1754 } else { 1755 // we can't see the whole thing, but what if there's a partial boundary? 1756 1757 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1758 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1759 bool potentialBoundaryFound = false; 1760 1761 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1762 // we grow the boundary a bit each time. If we think it looks the 1763 // same, better pull another chunk to be sure it's not the end. 1764 // Starting small because exiting the loop early is desirable, since 1765 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1766 // the best we can do. 1767 if(a > pps.buffer.length) 1768 break; // FIXME: is this right? 1769 assert(a <= pps.buffer.length); 1770 assert(a > 0); 1771 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1772 // ok, there *might* be a boundary here, so let's 1773 // not treat the end as data yet. The rest is good to 1774 // use though, since if there was a boundary there, we'd 1775 // have handled it up above after locationOf. 1776 1777 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1778 consume(pps.buffer.length - a); 1779 pieceHasNewContent(); 1780 potentialBoundaryFound = true; 1781 break boundaryCheck; 1782 } 1783 } 1784 1785 if(!potentialBoundaryFound) { 1786 // we can consume the whole thing 1787 pps.piece.content ~= pps.buffer; 1788 pieceHasNewContent(); 1789 consume(pps.buffer.length); 1790 } else { 1791 // we found a possible boundary, but there was 1792 // insufficient data to be sure. 1793 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1794 1795 return; // wait for the next chunk. 1796 } 1797 } 1798 } 1799 } while(pps.buffer.length); 1800 1801 // btw all boundaries except the first should have a \r\n before them 1802 } else { 1803 // application/x-www-form-urlencoded and application/json 1804 1805 // not using maxContentLength because that might be cranked up to allow 1806 // large file uploads. We can handle them, but a huge post[] isn't any good. 1807 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1808 throw new Exception("wtf is up with such a gigantic form submission????"); 1809 1810 pps.buffer ~= chunk; 1811 1812 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1813 if(pps.buffer.length == pps.expectedLength) { 1814 if(pps.needsSavedBody) 1815 pps.postBody = cast(string) pps.buffer; 1816 else 1817 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1818 version(preserveData) 1819 originalPostData = pps.buffer; 1820 } else { 1821 // just for debugging 1822 } 1823 } 1824 } 1825 1826 protected void cleanUpPostDataState() { 1827 pps = PostParserState.init; 1828 } 1829 1830 /// you can override this function to somehow react 1831 /// to an upload in progress. 1832 /// 1833 /// Take note that parts of the CGI object is not yet 1834 /// initialized! Stuff from HTTP headers, including get[], is usable. 1835 /// But, none of post[] is usable, and you cannot write here. That's 1836 /// why this method is const - mutating the object won't do much anyway. 1837 /// 1838 /// My idea here was so you can output a progress bar or 1839 /// something to a cooperative client (see arsd.rtud for a potential helper) 1840 /// 1841 /// The default is to do nothing. Subclass cgi and use the 1842 /// CustomCgiMain mixin to do something here. 1843 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1844 // This space intentionally left blank. 1845 } 1846 1847 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1848 /// *closeConnection will be set to true if you should close the connection after handling this request 1849 this(BufferedInputRange ir, bool* closeConnection) { 1850 isCalledWithCommandLineArguments = false; 1851 import al = std.algorithm; 1852 1853 immutable(ubyte)[] data; 1854 1855 void rdo(const(ubyte)[] d) { 1856 //import std.stdio; writeln(d); 1857 sendAll(ir.source, d); 1858 } 1859 1860 auto ira = ir.source.remoteAddress(); 1861 auto irLocalAddress = ir.source.localAddress(); 1862 1863 ushort port = 80; 1864 if(auto ia = cast(InternetAddress) irLocalAddress) { 1865 port = ia.port; 1866 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1867 port = ia.port; 1868 } 1869 1870 // that check for UnixAddress is to work around a Phobos bug 1871 // see: https://github.com/dlang/phobos/pull/7383 1872 // but this might be more useful anyway tbh for this case 1873 version(Posix) 1874 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1875 else 1876 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1877 } 1878 1879 /** 1880 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1881 1882 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1883 1884 Params: 1885 inputData = the incoming data, including headers and other raw http data. 1886 When the constructor exits, it will leave this range exactly at the start of 1887 the next request on the connection (if there is one). 1888 1889 address = the IP address of the remote user 1890 _port = the port number of the connection 1891 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1892 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1893 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1894 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1895 closeConnection = if the request asks to close the connection, *closeConnection == true. 1896 */ 1897 this( 1898 BufferedInputRange inputData, 1899 // string[] headers, immutable(ubyte)[] data, 1900 string address, ushort _port, 1901 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1902 bool _https = false, 1903 void delegate(const(ubyte)[]) _rawDataOutput = null, 1904 void delegate() _flush = null, 1905 // this pointer tells if the connection is supposed to be closed after we handle this 1906 bool* closeConnection = null) 1907 { 1908 // these are all set locally so the loop works 1909 // without triggering errors in dmd 2.064 1910 // we go ahead and set them at the end of it to the this version 1911 int port; 1912 string referrer; 1913 string remoteAddress; 1914 string userAgent; 1915 string authorization; 1916 string origin; 1917 string accept; 1918 string lastEventId; 1919 bool https; 1920 string host; 1921 RequestMethod requestMethod; 1922 string requestUri; 1923 string pathInfo; 1924 string queryString; 1925 string scriptName; 1926 string[string] get; 1927 string[][string] getArray; 1928 bool keepAliveRequested; 1929 bool acceptsGzip; 1930 string cookie; 1931 1932 1933 1934 environmentVariables = cast(const) environment.toAA; 1935 1936 idlol = inputData; 1937 1938 isCalledWithCommandLineArguments = false; 1939 1940 https = _https; 1941 port = _port; 1942 1943 rawDataOutput = _rawDataOutput; 1944 flushDelegate = _flush; 1945 nph = true; 1946 1947 remoteAddress = address; 1948 1949 // streaming parser 1950 import al = std.algorithm; 1951 1952 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1953 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1954 while(idx == -1) { 1955 inputData.popFront(0); 1956 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1957 } 1958 1959 assert(idx != -1); 1960 1961 1962 string contentType = ""; 1963 string[string] requestHeadersHere; 1964 1965 size_t contentLength; 1966 1967 bool isChunked; 1968 1969 { 1970 import core.runtime; 1971 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 1972 } 1973 1974 1975 int headerNumber = 0; 1976 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 1977 if(line.length) { 1978 headerNumber++; 1979 auto header = cast(string) line.idup; 1980 if(headerNumber == 1) { 1981 // request line 1982 auto parts = al.splitter(header, " "); 1983 if(parts.front == "PRI") { 1984 // this is an HTTP/2.0 line - "PRI * HTTP/2.0" - which indicates their payload will follow 1985 // we're going to immediately refuse this, im not interested in implementing http2 (it is unlikely 1986 // to bring me benefit) 1987 throw new HttpVersionNotSupportedException(); 1988 } 1989 requestMethod = to!RequestMethod(parts.front); 1990 parts.popFront(); 1991 requestUri = parts.front; 1992 1993 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 1994 scriptName = requestUri[0 .. pathInfoStarts]; 1995 1996 auto question = requestUri.indexOf("?"); 1997 if(question == -1) { 1998 queryString = ""; 1999 // FIXME: double check, this might be wrong since it could be url encoded 2000 pathInfo = requestUri[pathInfoStarts..$]; 2001 } else { 2002 queryString = requestUri[question+1..$]; 2003 pathInfo = requestUri[pathInfoStarts..question]; 2004 } 2005 2006 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 2007 getArray = cast(string[][string]) assumeUnique(ugh); 2008 2009 if(header.indexOf("HTTP/1.0") != -1) { 2010 http10 = true; 2011 autoBuffer = true; 2012 if(closeConnection) { 2013 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 2014 *closeConnection = true; 2015 } 2016 } 2017 } else { 2018 // other header 2019 auto colon = header.indexOf(":"); 2020 if(colon == -1) 2021 throw new Exception("HTTP headers should have a colon!"); 2022 string name = header[0..colon].toLower; 2023 string value = header[colon+2..$]; // skip the colon and the space 2024 2025 requestHeadersHere[name] = value; 2026 2027 if (name == "accept") { 2028 accept = value; 2029 } 2030 else if (name == "origin") { 2031 origin = value; 2032 } 2033 else if (name == "connection") { 2034 if(value == "close" && closeConnection) 2035 *closeConnection = true; 2036 if(value.asLowerCase().canFind("keep-alive")) { 2037 keepAliveRequested = true; 2038 2039 // on http 1.0, the connection is closed by default, 2040 // but not if they request keep-alive. then we don't close 2041 // anymore - undoing the set above 2042 if(http10 && closeConnection) { 2043 *closeConnection = false; 2044 } 2045 } 2046 } 2047 else if (name == "transfer-encoding") { 2048 if(value == "chunked") 2049 isChunked = true; 2050 } 2051 else if (name == "last-event-id") { 2052 lastEventId = value; 2053 } 2054 else if (name == "authorization") { 2055 authorization = value; 2056 } 2057 else if (name == "content-type") { 2058 contentType = value; 2059 } 2060 else if (name == "content-length") { 2061 contentLength = to!size_t(value); 2062 } 2063 else if (name == "x-forwarded-for") { 2064 remoteAddress = value; 2065 } 2066 else if (name == "x-forwarded-host" || name == "host") { 2067 if(name != "host" || host is null) 2068 host = value; 2069 } 2070 // FIXME: https://tools.ietf.org/html/rfc7239 2071 else if (name == "accept-encoding") { 2072 if(value.indexOf("gzip") != -1) 2073 acceptsGzip = true; 2074 } 2075 else if (name == "user-agent") { 2076 userAgent = value; 2077 } 2078 else if (name == "referer") { 2079 referrer = value; 2080 } 2081 else if (name == "cookie") { 2082 cookie ~= value; 2083 } else if(name == "expect") { 2084 if(value == "100-continue") { 2085 // FIXME we should probably give user code a chance 2086 // to process and reject but that needs to be virtual, 2087 // perhaps part of the CGI redesign. 2088 2089 // FIXME: if size is > max content length it should 2090 // also fail at this point. 2091 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 2092 2093 // FIXME: let the user write out 103 early hints too 2094 } 2095 } 2096 // else 2097 // ignore it 2098 2099 } 2100 } 2101 2102 inputData.consume(idx + 4); 2103 // done 2104 2105 requestHeaders = assumeUnique(requestHeadersHere); 2106 2107 ByChunkRange dataByChunk; 2108 2109 // reading Content-Length type data 2110 // We need to read up the data we have, and write it out as a chunk. 2111 if(!isChunked) { 2112 dataByChunk = byChunk(inputData, contentLength); 2113 } else { 2114 // chunked requests happen, but not every day. Since we need to know 2115 // the content length (for now, maybe that should change), we'll buffer 2116 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 2117 auto data = dechunk(inputData); 2118 2119 // set the range here 2120 dataByChunk = byChunk(data); 2121 contentLength = data.length; 2122 } 2123 2124 assert(dataByChunk !is null); 2125 2126 if(contentLength) { 2127 prepareForIncomingDataChunks(contentType, contentLength); 2128 foreach(dataChunk; dataByChunk) { 2129 handleIncomingDataChunk(dataChunk); 2130 } 2131 postArray = assumeUnique(pps._post); 2132 filesArray = assumeUnique(pps._files); 2133 files = keepLastOf(filesArray); 2134 post = keepLastOf(postArray); 2135 postBody = pps.postBody; 2136 cleanUpPostDataState(); 2137 } 2138 2139 this.port = port; 2140 this.referrer = referrer; 2141 this.remoteAddress = remoteAddress; 2142 this.userAgent = userAgent; 2143 this.authorization = authorization; 2144 this.origin = origin; 2145 this.accept = accept; 2146 this.lastEventId = lastEventId; 2147 this.https = https; 2148 this.host = host; 2149 this.requestMethod = requestMethod; 2150 this.requestUri = requestUri; 2151 this.pathInfo = pathInfo; 2152 this.queryString = queryString; 2153 2154 this.scriptName = scriptName; 2155 this.get = keepLastOf(getArray); 2156 this.getArray = cast(immutable) getArray; 2157 this.keepAliveRequested = keepAliveRequested; 2158 this.acceptsGzip = acceptsGzip; 2159 this.cookie = cookie; 2160 2161 cookiesArray = getCookieArray(); 2162 cookies = keepLastOf(cookiesArray); 2163 2164 } 2165 BufferedInputRange idlol; 2166 2167 private immutable(string[string]) keepLastOf(in string[][string] arr) { 2168 string[string] ca; 2169 foreach(k, v; arr) 2170 ca[k] = v[$-1]; 2171 2172 return assumeUnique(ca); 2173 } 2174 2175 // FIXME duplication 2176 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 2177 UploadedFile[string] ca; 2178 foreach(k, v; arr) 2179 ca[k] = v[$-1]; 2180 2181 return assumeUnique(ca); 2182 } 2183 2184 2185 private immutable(string[][string]) getCookieArray() { 2186 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 2187 return assumeUnique(forTheLoveOfGod); 2188 } 2189 2190 /// Very simple method to require a basic auth username and password. 2191 /// If the http request doesn't include the required credentials, it throws a 2192 /// HTTP 401 error, and an exception. 2193 /// 2194 /// Note: basic auth does not provide great security, especially over unencrypted HTTP; 2195 /// the user's credentials are sent in plain text on every request. 2196 /// 2197 /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 2198 /// application. Either use Apache's built in methods for basic authentication, or add 2199 /// something along these lines to your server configuration: 2200 /// 2201 /// RewriteEngine On 2202 /// RewriteCond %{HTTP:Authorization} ^(.*) 2203 /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 2204 /// 2205 /// To ensure the necessary data is available to cgi.d. 2206 void requireBasicAuth(string user, string pass, string message = null) { 2207 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 2208 setResponseStatus("401 Authorization Required"); 2209 header ("WWW-Authenticate: Basic realm=\""~message~"\""); 2210 close(); 2211 throw new Exception("Not authorized; got " ~ authorization); 2212 } 2213 } 2214 2215 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 2216 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 2217 /// Use setResponseExpires and updateResponseExpires for more control 2218 void setCache(bool allowCaching) { 2219 noCache = !allowCaching; 2220 } 2221 2222 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2223 /// who can accept it 2224 bool gzipResponse; 2225 2226 immutable bool acceptsGzip; 2227 immutable bool keepAliveRequested; 2228 2229 /// Set to true if and only if this was initialized with command line arguments 2230 immutable bool isCalledWithCommandLineArguments; 2231 2232 /// This gets a full url for the current request, including port, protocol, host, path, and query 2233 string getCurrentCompleteUri() const { 2234 ushort defaultPort = https ? 443 : 80; 2235 2236 string uri = "http"; 2237 if(https) 2238 uri ~= "s"; 2239 uri ~= "://"; 2240 uri ~= host; 2241 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2242 version(none) 2243 if(!(!port || port == defaultPort)) { 2244 uri ~= ":"; 2245 uri ~= to!string(port); 2246 } 2247 +/ 2248 uri ~= requestUri; 2249 return uri; 2250 } 2251 2252 /// You can override this if your site base url isn't the same as the script name 2253 string logicalScriptName() const { 2254 return scriptName; 2255 } 2256 2257 /++ 2258 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2259 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2260 Note setResponseStatus() must be called *before* you write() any data to the output. 2261 2262 History: 2263 The `int` overload was added on January 11, 2021. 2264 +/ 2265 void setResponseStatus(string status) { 2266 assert(!outputtedResponseData); 2267 responseStatus = status; 2268 } 2269 /// ditto 2270 void setResponseStatus(int statusCode) { 2271 setResponseStatus(getHttpCodeText(statusCode)); 2272 } 2273 private string responseStatus = null; 2274 2275 /// Returns true if it is still possible to output headers 2276 bool canOutputHeaders() { 2277 return !isClosed && !outputtedResponseData; 2278 } 2279 2280 /// Sets the location header, which the browser will redirect the user to automatically. 2281 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2282 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2283 void setResponseLocation(string uri, bool important = true, string status = null) { 2284 if(!important && isCurrentResponseLocationImportant) 2285 return; // important redirects always override unimportant ones 2286 2287 if(uri is null) { 2288 responseStatus = "200 OK"; 2289 responseLocation = null; 2290 isCurrentResponseLocationImportant = important; 2291 return; // this just cancels the redirect 2292 } 2293 2294 assert(!outputtedResponseData); 2295 if(status is null) 2296 responseStatus = "302 Found"; 2297 else 2298 responseStatus = status; 2299 2300 responseLocation = uri.strip; 2301 isCurrentResponseLocationImportant = important; 2302 } 2303 protected string responseLocation = null; 2304 private bool isCurrentResponseLocationImportant = false; 2305 2306 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2307 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2308 /// Note: the when parameter is different than setCookie's expire parameter. 2309 void setResponseExpires(long when, bool isPublic = false) { 2310 responseExpires = when; 2311 setCache(true); // need to enable caching so the date has meaning 2312 2313 responseIsPublic = isPublic; 2314 responseExpiresRelative = false; 2315 } 2316 2317 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2318 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2319 responseExpires = whenFromNow; 2320 setCache(true); // need to enable caching so the date has meaning 2321 2322 responseIsPublic = isPublic; 2323 responseExpiresRelative = true; 2324 } 2325 private long responseExpires = long.min; 2326 private bool responseIsPublic = false; 2327 private bool responseExpiresRelative = false; 2328 2329 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2330 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2331 /// output as a whole is as cacheable as the least cachable part in the chain. 2332 2333 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2334 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2335 void updateResponseExpires(long when, bool isPublic) { 2336 if(responseExpires == long.min) 2337 setResponseExpires(when, isPublic); 2338 else if(when < responseExpires) 2339 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2340 } 2341 2342 /* 2343 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2344 /// Should generally be false if the user is logged in. It assumes private cache only. 2345 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2346 void setPublicCaching(bool allowPublicCaches) { 2347 publicCaching = allowPublicCaches; 2348 } 2349 private bool publicCaching = false; 2350 */ 2351 2352 /++ 2353 History: 2354 Added January 11, 2021 2355 +/ 2356 enum SameSitePolicy { 2357 Lax, 2358 Strict, 2359 None 2360 } 2361 2362 /++ 2363 Sets an HTTP cookie, automatically encoding the data to the correct string. 2364 expiresIn is how many milliseconds in the future the cookie will expire. 2365 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2366 Note setCookie() must be called *before* you write() any data to the output. 2367 2368 History: 2369 Parameter `sameSitePolicy` was added on January 11, 2021. 2370 +/ 2371 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2372 assert(!outputtedResponseData); 2373 string cookie = std.uri.encodeComponent(name) ~ "="; 2374 cookie ~= std.uri.encodeComponent(data); 2375 if(path !is null) 2376 cookie ~= "; path=" ~ path; 2377 // FIXME: should I just be using max-age here? (also in cache below) 2378 if(expiresIn != 0) 2379 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2380 if(domain !is null) 2381 cookie ~= "; domain=" ~ domain; 2382 if(secure == true) 2383 cookie ~= "; Secure"; 2384 if(httpOnly == true ) 2385 cookie ~= "; HttpOnly"; 2386 final switch(sameSitePolicy) { 2387 case SameSitePolicy.Lax: 2388 cookie ~= "; SameSite=Lax"; 2389 break; 2390 case SameSitePolicy.Strict: 2391 cookie ~= "; SameSite=Strict"; 2392 break; 2393 case SameSitePolicy.None: 2394 cookie ~= "; SameSite=None"; 2395 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2396 break; 2397 } 2398 2399 if(auto idx = name in cookieIndexes) { 2400 responseCookies[*idx] = cookie; 2401 } else { 2402 cookieIndexes[name] = responseCookies.length; 2403 responseCookies ~= cookie; 2404 } 2405 } 2406 private string[] responseCookies; 2407 private size_t[string] cookieIndexes; 2408 2409 /// Clears a previously set cookie with the given name, path, and domain. 2410 void clearCookie(string name, string path = null, string domain = null) { 2411 assert(!outputtedResponseData); 2412 setCookie(name, "", 1, path, domain); 2413 } 2414 2415 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2416 void setResponseContentType(string ct) { 2417 assert(!outputtedResponseData); 2418 responseContentType = ct; 2419 } 2420 private string responseContentType = null; 2421 2422 /// Adds a custom header. It should be the name: value, but without any line terminator. 2423 /// For example: header("X-My-Header: Some value"); 2424 /// Note you should use the specialized functions in this object if possible to avoid 2425 /// duplicates in the output. 2426 void header(string h) { 2427 customHeaders ~= h; 2428 } 2429 2430 /++ 2431 I named the original function `header` after PHP, but this pattern more fits 2432 the rest of the Cgi object. 2433 2434 Either name are allowed. 2435 2436 History: 2437 Alias added June 17, 2022. 2438 +/ 2439 alias setResponseHeader = header; 2440 2441 private string[] customHeaders; 2442 private bool websocketMode; 2443 2444 void flushHeaders(const(void)[] t, bool isAll = false) { 2445 StackBuffer buffer = StackBuffer(0); 2446 2447 prepHeaders(t, isAll, &buffer); 2448 2449 if(rawDataOutput !is null) 2450 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2451 else { 2452 stdout.rawWrite(buffer.get()); 2453 } 2454 } 2455 2456 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2457 string terminator = "\n"; 2458 if(rawDataOutput !is null) 2459 terminator = "\r\n"; 2460 2461 if(responseStatus !is null) { 2462 if(nph) { 2463 if(http10) 2464 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2465 else 2466 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2467 } else 2468 buffer.add("Status: ", responseStatus, terminator); 2469 } else if (nph) { 2470 if(http10) 2471 buffer.add("HTTP/1.0 200 OK", terminator); 2472 else 2473 buffer.add("HTTP/1.1 200 OK", terminator); 2474 } 2475 2476 if(websocketMode) 2477 goto websocket; 2478 2479 if(nph) { // we're responsible for setting the date too according to http 1.1 2480 char[29] db = void; 2481 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2482 buffer.add("Date: ", db[], terminator); 2483 } 2484 2485 // FIXME: what if the user wants to set his own content-length? 2486 // The custom header function can do it, so maybe that's best. 2487 // Or we could reuse the isAll param. 2488 if(responseLocation !is null) { 2489 buffer.add("Location: ", responseLocation, terminator); 2490 } 2491 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2492 if(responseExpiresRelative) { 2493 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2494 buffer.add(responseExpires); 2495 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2496 } else { 2497 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2498 char[29] db = void; 2499 printDateToBuffer(cast(DateTime) expires, db[]); 2500 buffer.add("Expires: ", db[], terminator); 2501 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2502 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2503 buffer.add(terminator); 2504 } 2505 } 2506 if(responseCookies !is null && responseCookies.length > 0) { 2507 foreach(c; responseCookies) 2508 buffer.add("Set-Cookie: ", c, terminator); 2509 } 2510 if(noCache) { // we specifically do not want caching (this is actually the default) 2511 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2512 buffer.add("Expires: 0", terminator); 2513 buffer.add("Pragma: no-cache", terminator); 2514 } else { 2515 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2516 buffer.add("Cache-Control: public", terminator); 2517 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2518 } 2519 } 2520 if(responseContentType !is null) { 2521 buffer.add("Content-Type: ", responseContentType, terminator); 2522 } else 2523 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2524 2525 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2526 buffer.add("Content-Encoding: gzip", terminator); 2527 } 2528 2529 2530 if(!isAll) { 2531 if(nph && !http10) { 2532 buffer.add("Transfer-Encoding: chunked", terminator); 2533 responseChunked = true; 2534 } 2535 } else { 2536 buffer.add("Content-Length: "); 2537 buffer.add(t.length); 2538 buffer.add(terminator); 2539 if(nph && keepAliveRequested) { 2540 buffer.add("Connection: Keep-Alive", terminator); 2541 } 2542 } 2543 2544 websocket: 2545 2546 foreach(hd; customHeaders) 2547 buffer.add(hd, terminator); 2548 2549 // FIXME: what about duplicated headers? 2550 2551 // end of header indicator 2552 buffer.add(terminator); 2553 2554 outputtedResponseData = true; 2555 } 2556 2557 /// Writes the data to the output, flushing headers if they have not yet been sent. 2558 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2559 assert(!closed, "Output has already been closed"); 2560 2561 StackBuffer buffer = StackBuffer(0); 2562 2563 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2564 // actually gzip the data here 2565 2566 auto c = new Compress(HeaderFormat.gzip); // want gzip 2567 2568 auto data = c.compress(t); 2569 data ~= c.flush(); 2570 2571 // std.file.write("/tmp/last-item", data); 2572 2573 t = data; 2574 } 2575 2576 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2577 prepHeaders(t, isAll, &buffer); 2578 } 2579 2580 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2581 if (autoBuffer && !isAll) { 2582 outputBuffer ~= cast(ubyte[]) t; 2583 } 2584 if(!autoBuffer || isAll) { 2585 if(rawDataOutput !is null) 2586 if(nph && responseChunked) { 2587 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2588 // we're making the chunk here instead of in a function 2589 // to avoid unneeded gc pressure 2590 buffer.add(toHex(t.length)); 2591 buffer.add("\r\n"); 2592 buffer.add(cast(char[]) t, "\r\n"); 2593 } else { 2594 buffer.add(cast(char[]) t); 2595 } 2596 else 2597 buffer.add(cast(char[]) t); 2598 } 2599 } 2600 2601 if(rawDataOutput !is null) 2602 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2603 else 2604 stdout.rawWrite(buffer.get()); 2605 2606 if(maybeAutoClose && isAll) 2607 close(); // if you say it is all, that means we're definitely done 2608 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2609 } 2610 2611 /++ 2612 Convenience method to set content type to json and write the string as the complete response. 2613 2614 History: 2615 Added January 16, 2020 2616 +/ 2617 void writeJson(string json) { 2618 this.setResponseContentType("application/json"); 2619 this.write(json, true); 2620 } 2621 2622 /// Flushes the pending buffer, leaving the connection open so you can send more. 2623 void flush() { 2624 if(rawDataOutput is null) 2625 stdout.flush(); 2626 else if(flushDelegate !is null) 2627 flushDelegate(); 2628 } 2629 2630 version(autoBuffer) 2631 bool autoBuffer = true; 2632 else 2633 bool autoBuffer = false; 2634 ubyte[] outputBuffer; 2635 2636 /// Flushes the buffers to the network, signifying that you are done. 2637 /// You should always call this explicitly when you are done outputting data. 2638 void close() { 2639 if(closed) 2640 return; // don't double close 2641 2642 if(!outputtedResponseData) 2643 write("", true, false); 2644 2645 // writing auto buffered data 2646 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2647 if(!nph) 2648 stdout.rawWrite(outputBuffer); 2649 else 2650 write(outputBuffer, true, false); // tell it this is everything 2651 } 2652 2653 // closing the last chunk... 2654 if(nph && rawDataOutput !is null && responseChunked) 2655 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2656 2657 if(flushDelegate) 2658 flushDelegate(); 2659 2660 closed = true; 2661 } 2662 2663 // Closes without doing anything, shouldn't be used often 2664 void rawClose() { 2665 closed = true; 2666 } 2667 2668 /++ 2669 Gets a request variable as a specific type, or the default value of it isn't there 2670 or isn't convertible to the request type. 2671 2672 Checks both GET and POST variables, preferring the POST variable, if available. 2673 2674 A nice trick is using the default value to choose the type: 2675 2676 --- 2677 /* 2678 The return value will match the type of the default. 2679 Here, I gave 10 as a default, so the return value will 2680 be an int. 2681 2682 If the user-supplied value cannot be converted to the 2683 requested type, you will get the default value back. 2684 */ 2685 int a = cgi.request("number", 10); 2686 2687 if(cgi.get["number"] == "11") 2688 assert(a == 11); // conversion succeeds 2689 2690 if("number" !in cgi.get) 2691 assert(a == 10); // no value means you can't convert - give the default 2692 2693 if(cgi.get["number"] == "twelve") 2694 assert(a == 10); // conversion from string to int would fail, so we get the default 2695 --- 2696 2697 You can use an enum as an easy whitelist, too: 2698 2699 --- 2700 enum Operations { 2701 add, remove, query 2702 } 2703 2704 auto op = cgi.request("op", Operations.query); 2705 2706 if(cgi.get["op"] == "add") 2707 assert(op == Operations.add); 2708 if(cgi.get["op"] == "remove") 2709 assert(op == Operations.remove); 2710 if(cgi.get["op"] == "query") 2711 assert(op == Operations.query); 2712 2713 if(cgi.get["op"] == "random string") 2714 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2715 --- 2716 +/ 2717 T request(T = string)(in string name, in T def = T.init) const nothrow { 2718 try { 2719 return 2720 (name in post) ? to!T(post[name]) : 2721 (name in get) ? to!T(get[name]) : 2722 def; 2723 } catch(Exception e) { return def; } 2724 } 2725 2726 /// Is the output already closed? 2727 bool isClosed() const { 2728 return closed; 2729 } 2730 2731 /++ 2732 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2733 +/ 2734 Session!Data getSessionObject(Data)() { 2735 if(testInProcess !is null) { 2736 // test mode 2737 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2738 if(obj !is null) 2739 return cast(typeof(return)) obj; 2740 else { 2741 auto o = new MockSession!Data(); 2742 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2743 return o; 2744 } 2745 } else { 2746 // normal operation 2747 return new BasicDataServerSession!Data(this); 2748 } 2749 } 2750 2751 // if it is in test mode; triggers mock sessions. Used by CgiTester 2752 version(with_breaking_cgi_features) 2753 private CgiTester testInProcess; 2754 2755 /* Hooks for redirecting input and output */ 2756 private void delegate(const(ubyte)[]) rawDataOutput = null; 2757 private void delegate() flushDelegate = null; 2758 2759 /* This info is used when handling a more raw HTTP protocol */ 2760 private bool nph; 2761 private bool http10; 2762 private bool closed; 2763 private bool responseChunked = false; 2764 2765 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2766 immutable(ubyte)[] originalPostData; 2767 2768 /++ 2769 This holds the posted body data if it has not been parsed into [post] and [postArray]. 2770 2771 It is intended to be used for JSON and XML request content types, but also may be used 2772 for other content types your application can handle. But it will NOT be populated 2773 for content types application/x-www-form-urlencoded or multipart/form-data, since those are 2774 parsed into the post and postArray members. 2775 2776 Remember that anything beyond your `maxContentLength` param when setting up [GenericMain], etc., 2777 will be discarded to the client with an error. This helps keep this array from being exploded in size 2778 and consuming all your server's memory (though it may still be possible to eat excess ram from a concurrent 2779 client in certain build modes.) 2780 2781 History: 2782 Added January 5, 2021 2783 Documented February 21, 2023 (dub v11.0) 2784 +/ 2785 public immutable string postBody; 2786 alias postJson = postBody; // old name 2787 2788 /* Internal state flags */ 2789 private bool outputtedResponseData; 2790 private bool noCache = true; 2791 2792 const(string[string]) environmentVariables; 2793 2794 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2795 partially because it logically is (your code doesn't change what the user requested...) 2796 and partially because I hate how bad programs in PHP change those superglobals to do 2797 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2798 2799 For some of these, you'll want to refer to the http or cgi specs for more details. 2800 */ 2801 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2802 2803 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2804 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2805 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2806 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2807 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2808 immutable(char[]) scriptFileName; /// The physical filename of your script 2809 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2810 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2811 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2812 2813 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2814 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2815 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2816 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2817 2818 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2819 */ 2820 immutable(char[]) referrer; 2821 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2822 2823 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2824 2825 immutable bool https; /// Was the request encrypted via https? 2826 immutable int port; /// On what TCP port number did the server receive the request? 2827 2828 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2829 2830 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2831 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2832 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2833 2834 /** 2835 Represents user uploaded files. 2836 2837 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2838 */ 2839 immutable(UploadedFile[][string]) filesArray; 2840 immutable(UploadedFile[string]) files; 2841 2842 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2843 /// the order of the arrays is the order the data arrives 2844 immutable(string[][string]) getArray; /// like get, but an array of values per name 2845 immutable(string[][string]) postArray; /// ditto for post 2846 immutable(string[][string]) cookiesArray; /// ditto for cookies 2847 2848 // convenience function for appending to a uri without extra ? 2849 // matches the name and effect of javascript's location.search property 2850 string search() const { 2851 if(queryString.length) 2852 return "?" ~ queryString; 2853 return ""; 2854 } 2855 2856 // FIXME: what about multiple files with the same name? 2857 private: 2858 //RequestMethod _requestMethod; 2859 } 2860 2861 /// use this for testing or other isolated things when you want it to be no-ops 2862 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2863 // we want to ignore, not use stdout 2864 if(outputSink is null) 2865 outputSink = delegate void(const(ubyte)[]) { }; 2866 2867 string[string] env; 2868 env["REQUEST_METHOD"] = to!string(method); 2869 env["CONTENT_LENGTH"] = to!string(data.length); 2870 2871 auto cgi = new Cgi( 2872 0, 2873 env, 2874 { return data; }, 2875 outputSink, 2876 null); 2877 2878 return cgi; 2879 } 2880 2881 /++ 2882 A helper test class for request handler unittests. 2883 +/ 2884 version(with_breaking_cgi_features) 2885 class CgiTester { 2886 private { 2887 SessionObject[TypeInfo] mockSessions; 2888 SessionObject getSessionOverride(TypeInfo ti) { 2889 if(auto o = ti in mockSessions) 2890 return *o; 2891 else 2892 return null; 2893 } 2894 void setSessionOverride(TypeInfo ti, SessionObject so) { 2895 mockSessions[ti] = so; 2896 } 2897 } 2898 2899 /++ 2900 Gets (and creates if necessary) a mock session object for this test. Note 2901 it will be the same one used for any test operations through this CgiTester instance. 2902 +/ 2903 Session!Data getSessionObject(Data)() { 2904 auto obj = getSessionOverride(typeid(typeof(return))); 2905 if(obj !is null) 2906 return cast(typeof(return)) obj; 2907 else { 2908 auto o = new MockSession!Data(); 2909 setSessionOverride(typeid(typeof(return)), o); 2910 return o; 2911 } 2912 } 2913 2914 /++ 2915 Pass a reference to your request handler when creating the tester. 2916 +/ 2917 this(void function(Cgi) requestHandler) { 2918 this.requestHandler = requestHandler; 2919 } 2920 2921 /++ 2922 You can check response information with these methods after you call the request handler. 2923 +/ 2924 struct Response { 2925 int code; 2926 string[string] headers; 2927 string responseText; 2928 ubyte[] responseBody; 2929 } 2930 2931 /++ 2932 Executes a test request on your request handler, and returns the response. 2933 2934 Params: 2935 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2936 args = additional arguments. Same format as cgi's command line handler. 2937 +/ 2938 Response GET(string url, string[] args = null) { 2939 return executeTest("GET", url, args); 2940 } 2941 /// ditto 2942 Response POST(string url, string[] args = null) { 2943 return executeTest("POST", url, args); 2944 } 2945 2946 /// ditto 2947 Response executeTest(string method, string url, string[] args) { 2948 ubyte[] outputtedRawData; 2949 void outputSink(const(ubyte)[] data) { 2950 outputtedRawData ~= data; 2951 } 2952 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2953 cgi.testInProcess = this; 2954 scope(exit) cgi.dispose(); 2955 2956 requestHandler(cgi); 2957 2958 cgi.close(); 2959 2960 Response response; 2961 2962 if(outputtedRawData.length) { 2963 enum LINE = "\r\n"; 2964 2965 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 2966 assert(idx != -1, to!string(outputtedRawData)); 2967 auto headers = cast(string) outputtedRawData[0 .. idx]; 2968 response.code = 200; 2969 while(headers.length) { 2970 auto i = headers.locationOf(LINE); 2971 if(i == -1) i = cast(int) headers.length; 2972 2973 auto header = headers[0 .. i]; 2974 2975 auto c = header.locationOf(":"); 2976 if(c != -1) { 2977 auto name = header[0 .. c]; 2978 auto value = header[c + 2 ..$]; 2979 2980 if(name == "Status") 2981 response.code = value[0 .. value.locationOf(" ")].to!int; 2982 2983 response.headers[name] = value; 2984 } else { 2985 assert(0); 2986 } 2987 2988 if(i != headers.length) 2989 i += 2; 2990 headers = headers[i .. $]; 2991 } 2992 response.responseBody = outputtedRawData[idx + 4 .. $]; 2993 response.responseText = cast(string) response.responseBody; 2994 } 2995 2996 return response; 2997 } 2998 2999 private void function(Cgi) requestHandler; 3000 } 3001 3002 3003 // should this be a separate module? Probably, but that's a hassle. 3004 3005 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 3006 string makeDataUrl(string mimeType, in void[] data) { 3007 auto data64 = Base64.encode(cast(const(ubyte[])) data); 3008 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 3009 } 3010 3011 // FIXME: I don't think this class correctly decodes/encodes the individual parts 3012 /// Represents a url that can be broken down or built up through properties 3013 struct Uri { 3014 alias toString this; // blargh idk a url really is a string, but should it be implicit? 3015 3016 // scheme//userinfo@host:port/path?query#fragment 3017 3018 string scheme; /// e.g. "http" in "http://example.com/" 3019 string userinfo; /// the username (and possibly a password) in the uri 3020 string host; /// the domain name 3021 int port; /// port number, if given. Will be zero if a port was not explicitly given 3022 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 3023 string query; /// the stuff after the ? in a uri 3024 string fragment; /// the stuff after the # in a uri. 3025 3026 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 3027 // the decode ones need to keep different names anyway because we can't overload on return values... 3028 static string encode(string s) { return std.uri.encodeComponent(s); } 3029 static string encode(string[string] s) { return encodeVariables(s); } 3030 static string encode(string[][string] s) { return encodeVariables(s); } 3031 3032 /// Breaks down a uri string to its components 3033 this(string uri) { 3034 reparse(uri); 3035 } 3036 3037 private void reparse(string uri) { 3038 // from RFC 3986 3039 // the ctRegex triples the compile time and makes ugly errors for no real benefit 3040 // it was a nice experiment but just not worth it. 3041 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 3042 /* 3043 Captures: 3044 0 = whole url 3045 1 = scheme, with : 3046 2 = scheme, no : 3047 3 = authority, with // 3048 4 = authority, no // 3049 5 = path 3050 6 = query string, with ? 3051 7 = query string, no ? 3052 8 = anchor, with # 3053 9 = anchor, no # 3054 */ 3055 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 3056 // instead, I will DIY and cut that down to 0.6s on the same computer. 3057 /* 3058 3059 Note that authority is 3060 user:password@domain:port 3061 where the user:password@ part is optional, and the :port is optional. 3062 3063 Regex translation: 3064 3065 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 3066 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 3067 Path cannot have any ? or # in it. It is optional. 3068 Query must start with ? and must not have # in it. It is optional. 3069 Anchor must start with # and can have anything else in it to end of string. It is optional. 3070 */ 3071 3072 this = Uri.init; // reset all state 3073 3074 // empty uri = nothing special 3075 if(uri.length == 0) { 3076 return; 3077 } 3078 3079 size_t idx; 3080 3081 scheme_loop: foreach(char c; uri[idx .. $]) { 3082 switch(c) { 3083 case ':': 3084 case '/': 3085 case '?': 3086 case '#': 3087 break scheme_loop; 3088 default: 3089 } 3090 idx++; 3091 } 3092 3093 if(idx == 0 && uri[idx] == ':') { 3094 // this is actually a path! we skip way ahead 3095 goto path_loop; 3096 } 3097 3098 if(idx == uri.length) { 3099 // the whole thing is a path, apparently 3100 path = uri; 3101 return; 3102 } 3103 3104 if(idx > 0 && uri[idx] == ':') { 3105 scheme = uri[0 .. idx]; 3106 idx++; 3107 } else { 3108 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 3109 idx = 0; 3110 } 3111 3112 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 3113 // we have an authority.... 3114 idx += 2; 3115 3116 auto authority_start = idx; 3117 authority_loop: foreach(char c; uri[idx .. $]) { 3118 switch(c) { 3119 case '/': 3120 case '?': 3121 case '#': 3122 break authority_loop; 3123 default: 3124 } 3125 idx++; 3126 } 3127 3128 auto authority = uri[authority_start .. idx]; 3129 3130 auto idx2 = authority.indexOf("@"); 3131 if(idx2 != -1) { 3132 userinfo = authority[0 .. idx2]; 3133 authority = authority[idx2 + 1 .. $]; 3134 } 3135 3136 if(authority.length && authority[0] == '[') { 3137 // ipv6 address special casing 3138 idx2 = authority.indexOf(']'); 3139 if(idx2 != -1) { 3140 auto end = authority[idx2 + 1 .. $]; 3141 if(end.length && end[0] == ':') 3142 idx2 = idx2 + 1; 3143 else 3144 idx2 = -1; 3145 } 3146 } else { 3147 idx2 = authority.indexOf(":"); 3148 } 3149 3150 if(idx2 == -1) { 3151 port = 0; // 0 means not specified; we should use the default for the scheme 3152 host = authority; 3153 } else { 3154 host = authority[0 .. idx2]; 3155 if(idx2 + 1 < authority.length) 3156 port = to!int(authority[idx2 + 1 .. $]); 3157 else 3158 port = 0; 3159 } 3160 } 3161 3162 path_loop: 3163 auto path_start = idx; 3164 3165 foreach(char c; uri[idx .. $]) { 3166 if(c == '?' || c == '#') 3167 break; 3168 idx++; 3169 } 3170 3171 path = uri[path_start .. idx]; 3172 3173 if(idx == uri.length) 3174 return; // nothing more to examine... 3175 3176 if(uri[idx] == '?') { 3177 idx++; 3178 auto query_start = idx; 3179 foreach(char c; uri[idx .. $]) { 3180 if(c == '#') 3181 break; 3182 idx++; 3183 } 3184 query = uri[query_start .. idx]; 3185 } 3186 3187 if(idx < uri.length && uri[idx] == '#') { 3188 idx++; 3189 fragment = uri[idx .. $]; 3190 } 3191 3192 // uriInvalidated = false; 3193 } 3194 3195 private string rebuildUri() const { 3196 string ret; 3197 if(scheme.length) 3198 ret ~= scheme ~ ":"; 3199 if(userinfo.length || host.length) 3200 ret ~= "//"; 3201 if(userinfo.length) 3202 ret ~= userinfo ~ "@"; 3203 if(host.length) 3204 ret ~= host; 3205 if(port) 3206 ret ~= ":" ~ to!string(port); 3207 3208 ret ~= path; 3209 3210 if(query.length) 3211 ret ~= "?" ~ query; 3212 3213 if(fragment.length) 3214 ret ~= "#" ~ fragment; 3215 3216 // uri = ret; 3217 // uriInvalidated = false; 3218 return ret; 3219 } 3220 3221 /// Converts the broken down parts back into a complete string 3222 string toString() const { 3223 // if(uriInvalidated) 3224 return rebuildUri(); 3225 } 3226 3227 /// Returns a new absolute Uri given a base. It treats this one as 3228 /// relative where possible, but absolute if not. (If protocol, domain, or 3229 /// other info is not set, the new one inherits it from the base.) 3230 /// 3231 /// Browsers use a function like this to figure out links in html. 3232 Uri basedOn(in Uri baseUrl) const { 3233 Uri n = this; // copies 3234 if(n.scheme == "data") 3235 return n; 3236 // n.uriInvalidated = true; // make sure we regenerate... 3237 3238 // userinfo is not inherited... is this wrong? 3239 3240 // if anything is given in the existing url, we don't use the base anymore. 3241 if(n.scheme.empty) { 3242 n.scheme = baseUrl.scheme; 3243 if(n.host.empty) { 3244 n.host = baseUrl.host; 3245 if(n.port == 0) { 3246 n.port = baseUrl.port; 3247 if(n.path.length > 0 && n.path[0] != '/') { 3248 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3249 if(b.length == 0) 3250 b = "/"; 3251 n.path = b ~ n.path; 3252 } else if(n.path.length == 0) { 3253 n.path = baseUrl.path; 3254 } 3255 } 3256 } 3257 } 3258 3259 n.removeDots(); 3260 3261 return n; 3262 } 3263 3264 void removeDots() { 3265 auto parts = this.path.split("/"); 3266 string[] toKeep; 3267 foreach(part; parts) { 3268 if(part == ".") { 3269 continue; 3270 } else if(part == "..") { 3271 //if(toKeep.length > 1) 3272 toKeep = toKeep[0 .. $-1]; 3273 //else 3274 //toKeep = [""]; 3275 continue; 3276 } else { 3277 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3278 //continue; // skip a `//` situation 3279 toKeep ~= part; 3280 } 3281 } 3282 3283 auto path = toKeep.join("/"); 3284 if(path.length && path[0] != '/') 3285 path = "/" ~ path; 3286 3287 this.path = path; 3288 } 3289 3290 unittest { 3291 auto uri = Uri("test.html"); 3292 assert(uri.path == "test.html"); 3293 uri = Uri("path/1/lol"); 3294 assert(uri.path == "path/1/lol"); 3295 uri = Uri("http://me@example.com"); 3296 assert(uri.scheme == "http"); 3297 assert(uri.userinfo == "me"); 3298 assert(uri.host == "example.com"); 3299 uri = Uri("http://example.com/#a"); 3300 assert(uri.scheme == "http"); 3301 assert(uri.host == "example.com"); 3302 assert(uri.fragment == "a"); 3303 uri = Uri("#foo"); 3304 assert(uri.fragment == "foo"); 3305 uri = Uri("?lol"); 3306 assert(uri.query == "lol"); 3307 uri = Uri("#foo?lol"); 3308 assert(uri.fragment == "foo?lol"); 3309 uri = Uri("?lol#foo"); 3310 assert(uri.fragment == "foo"); 3311 assert(uri.query == "lol"); 3312 3313 uri = Uri("http://127.0.0.1/"); 3314 assert(uri.host == "127.0.0.1"); 3315 assert(uri.port == 0); 3316 3317 uri = Uri("http://127.0.0.1:123/"); 3318 assert(uri.host == "127.0.0.1"); 3319 assert(uri.port == 123); 3320 3321 uri = Uri("http://[ff:ff::0]/"); 3322 assert(uri.host == "[ff:ff::0]"); 3323 3324 uri = Uri("http://[ff:ff::0]:123/"); 3325 assert(uri.host == "[ff:ff::0]"); 3326 assert(uri.port == 123); 3327 } 3328 3329 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3330 // the possibilities. 3331 unittest { 3332 auto url = Uri("cool.html"); // checking relative links 3333 3334 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3335 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3336 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3337 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3338 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3339 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3340 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3341 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3342 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3343 3344 url = Uri("/something/cool.html"); // same server, different path 3345 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3346 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3347 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3348 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3349 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3350 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3351 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3352 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3353 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3354 3355 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3356 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3357 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3358 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3359 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3360 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3361 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3362 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3363 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3364 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3365 3366 url = Uri("/test/bar"); 3367 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3368 assert(Uri("../").basedOn(url) == "/"); 3369 3370 url = Uri("http://example.com/"); 3371 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3372 3373 //auto uriBefore = url; 3374 url = Uri("#anchor"); // everything should remain the same except the anchor 3375 //uriBefore.anchor = "anchor"); 3376 //assert(url == uriBefore); 3377 3378 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3379 3380 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3381 3382 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3383 3384 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3385 3386 // FIXME: add something for port too 3387 } 3388 3389 // these are like javascript's location.search and location.hash 3390 string search() const { 3391 return query.length ? ("?" ~ query) : ""; 3392 } 3393 string hash() const { 3394 return fragment.length ? ("#" ~ fragment) : ""; 3395 } 3396 } 3397 3398 3399 /* 3400 for session, see web.d 3401 */ 3402 3403 /// breaks down a url encoded string 3404 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3405 auto vars = data.split(separator); 3406 string[][string] _get; 3407 foreach(var; vars) { 3408 auto equal = var.indexOf("="); 3409 string name; 3410 string value; 3411 if(equal == -1) { 3412 name = decodeComponent(var); 3413 value = ""; 3414 } else { 3415 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3416 // stupid + -> space conversion. 3417 name = decodeComponent(var[0..equal].replace("+", " ")); 3418 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3419 } 3420 3421 _get[name] ~= value; 3422 if(namesInOrder) 3423 (*namesInOrder) ~= name; 3424 if(valuesInOrder) 3425 (*valuesInOrder) ~= value; 3426 } 3427 return _get; 3428 } 3429 3430 /// breaks down a url encoded string, but only returns the last value of any array 3431 string[string] decodeVariablesSingle(string data) { 3432 string[string] va; 3433 auto varArray = decodeVariables(data); 3434 foreach(k, v; varArray) 3435 va[k] = v[$-1]; 3436 3437 return va; 3438 } 3439 3440 /// url encodes the whole string 3441 string encodeVariables(in string[string] data) { 3442 string ret; 3443 3444 bool outputted = false; 3445 foreach(k, v; data) { 3446 if(outputted) 3447 ret ~= "&"; 3448 else 3449 outputted = true; 3450 3451 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3452 } 3453 3454 return ret; 3455 } 3456 3457 /// url encodes a whole string 3458 string encodeVariables(in string[][string] data) { 3459 string ret; 3460 3461 bool outputted = false; 3462 foreach(k, arr; data) { 3463 foreach(v; arr) { 3464 if(outputted) 3465 ret ~= "&"; 3466 else 3467 outputted = true; 3468 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3469 } 3470 } 3471 3472 return ret; 3473 } 3474 3475 /// Encodes all but the explicitly unreserved characters per rfc 3986 3476 /// Alphanumeric and -_.~ are the only ones left unencoded 3477 /// name is borrowed from php 3478 string rawurlencode(in char[] data) { 3479 string ret; 3480 ret.reserve(data.length * 2); 3481 foreach(char c; data) { 3482 if( 3483 (c >= 'a' && c <= 'z') || 3484 (c >= 'A' && c <= 'Z') || 3485 (c >= '0' && c <= '9') || 3486 c == '-' || c == '_' || c == '.' || c == '~') 3487 { 3488 ret ~= c; 3489 } else { 3490 ret ~= '%'; 3491 // since we iterate on char, this should give us the octets of the full utf8 string 3492 ret ~= toHexUpper(c); 3493 } 3494 } 3495 3496 return ret; 3497 } 3498 3499 3500 // http helper functions 3501 3502 // for chunked responses (which embedded http does whenever possible) 3503 version(none) // this is moved up above to avoid making a copy of the data 3504 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3505 const(ubyte)[] ret; 3506 3507 ret = cast(const(ubyte)[]) toHex(data.length); 3508 ret ~= cast(const(ubyte)[]) "\r\n"; 3509 ret ~= data; 3510 ret ~= cast(const(ubyte)[]) "\r\n"; 3511 3512 return ret; 3513 } 3514 3515 string toHex(long num) { 3516 string ret; 3517 while(num) { 3518 int v = num % 16; 3519 num /= 16; 3520 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3521 ret ~= d; 3522 } 3523 3524 return to!string(array(ret.retro)); 3525 } 3526 3527 string toHexUpper(long num) { 3528 string ret; 3529 while(num) { 3530 int v = num % 16; 3531 num /= 16; 3532 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3533 ret ~= d; 3534 } 3535 3536 if(ret.length == 1) 3537 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3538 3539 return to!string(array(ret.retro)); 3540 } 3541 3542 3543 // the generic mixins 3544 3545 /++ 3546 Use this instead of writing your own main 3547 3548 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3549 +/ 3550 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3551 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3552 } 3553 3554 /++ 3555 Boilerplate mixin for a main function that uses the [dispatcher] function. 3556 3557 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3558 3559 History: 3560 Added July 9, 2021 3561 +/ 3562 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3563 /// forwards to [CustomCgiDispatcherMain] with default args 3564 mixin CustomCgiDispatcherMain!(Cgi, defaultMaxContentLength, Presenter, DispatcherArgs); 3565 } 3566 3567 /// ditto 3568 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3569 class GenericPresenter : WebPresenter!GenericPresenter {} 3570 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3571 } 3572 3573 /++ 3574 Allows for a generic [DispatcherMain] with custom arguments. Note you can use [defaultMaxContentLength] as the second argument if you like. 3575 3576 History: 3577 Added May 13, 2023 (dub v11.0) 3578 +/ 3579 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, Presenter, DispatcherArgs...) { 3580 /++ 3581 Handler to the generated presenter you can use from your objects, etc. 3582 +/ 3583 Presenter activePresenter; 3584 3585 /++ 3586 Request handler that creates the presenter then forwards to the [dispatcher] function. 3587 Renders 404 if the dispatcher did not handle the request. 3588 3589 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3590 +/ 3591 void handler(Cgi cgi) { 3592 auto presenter = new Presenter; 3593 activePresenter = presenter; 3594 scope(exit) activePresenter = null; 3595 3596 if(cgi.dispatcher!DispatcherArgs(presenter)) 3597 return; 3598 3599 switch(cgi.pathInfo) { 3600 case "/style.css": 3601 cgi.setCache(true); 3602 cgi.setResponseContentType("text/css"); 3603 cgi.write(presenter.style(), true); 3604 break; 3605 case "/script.js": 3606 cgi.setCache(true); 3607 cgi.setResponseContentType("application/javascript"); 3608 cgi.write(presenter.script(), true); 3609 break; 3610 default: 3611 presenter.renderBasicError(cgi, 404); 3612 } 3613 } 3614 mixin CustomCgiMain!(CustomCgi, handler, maxContentLength); 3615 } 3616 3617 /// ditto 3618 mixin template CustomCgiDispatcherMain(CustomCgi, size_t maxContentLength, DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3619 class GenericPresenter : WebPresenter!GenericPresenter {} 3620 mixin CustomCgiDispatcherMain!(CustomCgi, maxContentLength, GenericPresenter, DispatcherArgs); 3621 3622 } 3623 3624 private string simpleHtmlEncode(string s) { 3625 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3626 } 3627 3628 string messageFromException(Throwable t) { 3629 string message; 3630 if(t !is null) { 3631 debug message = t.toString(); 3632 else message = "An unexpected error has occurred."; 3633 } else { 3634 message = "Unknown error"; 3635 } 3636 return message; 3637 } 3638 3639 string plainHttpError(bool isCgi, string type, Throwable t) { 3640 auto message = messageFromException(t); 3641 message = simpleHtmlEncode(message); 3642 3643 return format("%s %s\r\nContent-Length: %s\r\nConnection: close\r\n\r\n%s", 3644 isCgi ? "Status:" : "HTTP/1.1", 3645 type, message.length, message); 3646 } 3647 3648 // returns true if we were able to recover reasonably 3649 bool handleException(Cgi cgi, Throwable t) { 3650 if(cgi.isClosed) { 3651 // if the channel has been explicitly closed, we can't handle it here 3652 return true; 3653 } 3654 3655 if(cgi.outputtedResponseData) { 3656 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3657 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3658 } else { 3659 // no headers are sent, we can send a full blown error and recover 3660 cgi.setCache(false); 3661 cgi.setResponseContentType("text/html"); 3662 cgi.setResponseLocation(null); // cancel the redirect 3663 cgi.setResponseStatus("500 Internal Server Error"); 3664 cgi.write(simpleHtmlEncode(messageFromException(t))); 3665 cgi.close(); 3666 return true; 3667 } 3668 } 3669 3670 bool isCgiRequestMethod(string s) { 3671 s = s.toUpper(); 3672 if(s == "COMMANDLINE") 3673 return true; 3674 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3675 if(s == member) 3676 return true; 3677 return false; 3678 } 3679 3680 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3681 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3682 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3683 void main(string[] args) { 3684 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3685 } 3686 } 3687 3688 version(embedded_httpd_processes) 3689 __gshared int processPoolSize = 8; 3690 3691 // Returns true if run. You should exit the program after that. 3692 bool tryAddonServers(string[] args) { 3693 if(args.length > 1) { 3694 // run the special separate processes if needed 3695 switch(args[1]) { 3696 case "--websocket-server": 3697 version(with_addon_servers) 3698 websocketServers[args[2]](args[3 .. $]); 3699 else 3700 printf("Add-on servers not compiled in.\n"); 3701 return true; 3702 case "--websocket-servers": 3703 import core.demangle; 3704 version(with_addon_servers_connections) 3705 foreach(k, v; websocketServers) 3706 writeln(k, "\t", demangle(k)); 3707 return true; 3708 case "--session-server": 3709 version(with_addon_servers) 3710 runSessionServer(); 3711 else 3712 printf("Add-on servers not compiled in.\n"); 3713 return true; 3714 case "--event-server": 3715 version(with_addon_servers) 3716 runEventServer(); 3717 else 3718 printf("Add-on servers not compiled in.\n"); 3719 return true; 3720 case "--timer-server": 3721 version(with_addon_servers) 3722 runTimerServer(); 3723 else 3724 printf("Add-on servers not compiled in.\n"); 3725 return true; 3726 case "--timed-jobs": 3727 import core.demangle; 3728 version(with_addon_servers_connections) 3729 foreach(k, v; scheduledJobHandlers) 3730 writeln(k, "\t", demangle(k)); 3731 return true; 3732 case "--timed-job": 3733 scheduledJobHandlers[args[2]](args[3 .. $]); 3734 return true; 3735 default: 3736 // intentionally blank - do nothing and carry on to run normally 3737 } 3738 } 3739 return false; 3740 } 3741 3742 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3743 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3744 // we support command line thing for easy testing everywhere 3745 // it needs to be called ./app method uri [other args...] 3746 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3747 Cgi cgi = new CustomCgi(args); 3748 scope(exit) cgi.dispose(); 3749 fun(cgi); 3750 cgi.close(); 3751 return true; 3752 } 3753 return false; 3754 } 3755 3756 /++ 3757 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] for a complete, up-to-date, example of how it is used internally. 3758 3759 As of version 11 (released August 2023), you can also make things like this: 3760 3761 --- 3762 // listens on both a unix domain socket called `foo` and on the loopback interfaces port 8080 3763 RequestServer server = RequestServer(["http://unix:foo", "http://localhost:8080"]); 3764 3765 // can also: 3766 // RequestServer server = RequestServer(0); // listen on an OS-provided port on all interfaces 3767 3768 // NOT IMPLEMENTED YET 3769 // server.initialize(); // explicit initialization will populate any "any port" things and throw if a bind failed 3770 3771 foreach(listenSpec; server.listenSpecs) { 3772 // you can check what it actually bound to here and see your assigned ports 3773 } 3774 3775 // NOT IMPLEMENTED YET 3776 // server.start!handler(); // starts and runs in the arsd.core event loop 3777 3778 server.serve!handler(); // blocks the thread until the server exits 3779 --- 3780 3781 History: 3782 Added Sept 26, 2020 (release version 8.5). 3783 3784 The `listenSpec` member was added July 31, 2023. 3785 +/ 3786 struct RequestServer { 3787 /++ 3788 Sets the host and port the server will listen on. This is semi-deprecated; the new (as of July 31, 2023) [listenSpec] parameter obsoletes these. You cannot use both together; the listeningHost and listeningPort are ONLY used if listenSpec is null. 3789 +/ 3790 string listeningHost = defaultListeningHost(); 3791 /// ditto 3792 ushort listeningPort = defaultListeningPort(); 3793 3794 static struct ListenSpec { 3795 enum Protocol { 3796 http, 3797 https, 3798 scgi 3799 } 3800 Protocol protocol; 3801 3802 enum AddressType { 3803 ip, 3804 unix, 3805 abstract_ 3806 } 3807 AddressType addressType; 3808 3809 string address; 3810 ushort port; 3811 } 3812 3813 /++ 3814 The array of addresses you want to listen on. The format looks like a url but has a few differences. 3815 3816 This ONLY works on embedded_httpd_threads, embedded_httpd_hybrid, and scgi builds at this time. 3817 3818 `http://localhost:8080` 3819 3820 `http://unix:filename/here` 3821 3822 `scgi://abstract:/name/here` 3823 3824 `http://[::1]:4444` 3825 3826 Note that IPv6 addresses must be enclosed in brackets. If you want to listen on an interface called `unix` or `abstract`, contact me, that is not supported but I could add some kind of escape mechanism. 3827 3828 If you leave off the protocol, it assumes the default based on compile flags. If you only give a number, it is assumed to be a port on any tcp interface. 3829 3830 `localhost:8080` serves the default protocol. 3831 3832 `8080` or `:8080` assumes default protocol on localhost. 3833 3834 The protocols can be `http:`, `https:`, and `scgi:`. Original `cgi` is not supported with this, since it is transactional with a single process. 3835 3836 Valid hosts are an IPv4 address (with a mandatory port), an IPv6 address (with a mandatory port), just a port alone, `unix:/path/to/unix/socket` (which may be a relative path without a leading slash), or `abstract:/path/to/linux/abstract/namespace`. 3837 3838 `http://unix:foo` will serve http over the unix domain socket named `foo` in the current working directory. 3839 3840 $(PITFALL 3841 If you set this to anything non-null (including a non-null, zero-length array) any `listenSpec` entries, [listeningHost] and [listeningPort] are ignored. 3842 ) 3843 3844 Bugs: 3845 The implementation currently ignores the protocol spec in favor of the default compiled in option. 3846 3847 History: 3848 Added July 31, 2023 (dub v11.0) 3849 +/ 3850 string[] listenSpec; 3851 3852 /++ 3853 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3854 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3855 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3856 [stop] may not work as well. 3857 3858 History: 3859 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3860 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3861 compatibility. 3862 +/ 3863 bool useFork = cgi_use_fork_default; 3864 3865 /++ 3866 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3867 default based on the number of cpus modified by the server mode. 3868 3869 History: 3870 Added August 12, 2022 (dub v10.9) 3871 +/ 3872 int numberOfThreads = 0; 3873 3874 /++ 3875 Creates a server configured to listen to multiple URLs. 3876 3877 History: 3878 Added July 31, 2023 (dub v11.0) 3879 +/ 3880 this(string[] listenTo) { 3881 this.listenSpec = listenTo; 3882 } 3883 3884 /// Creates a server object configured to listen on a single host and port. 3885 this(string defaultHost, ushort defaultPort) { 3886 this.listeningHost = defaultHost; 3887 this.listeningPort = defaultPort; 3888 } 3889 3890 /// ditto 3891 this(ushort defaultPort) { 3892 listeningPort = defaultPort; 3893 } 3894 3895 /++ 3896 Reads the command line arguments into the values here. 3897 3898 Possible arguments are `--listen` (can appear multiple times), `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3899 3900 Please note you cannot combine `--listen` with `--listening-host` or `--listening-port` / `--port`. Use one or the other style. 3901 +/ 3902 void configureFromCommandLine(string[] args) { 3903 bool portOrHostFound = false; 3904 3905 bool foundPort = false; 3906 bool foundHost = false; 3907 bool foundUid = false; 3908 bool foundGid = false; 3909 bool foundListen = false; 3910 foreach(arg; args) { 3911 if(foundPort) { 3912 listeningPort = to!ushort(arg); 3913 portOrHostFound = true; 3914 foundPort = false; 3915 continue; 3916 } 3917 if(foundHost) { 3918 listeningHost = arg; 3919 portOrHostFound = true; 3920 foundHost = false; 3921 continue; 3922 } 3923 if(foundUid) { 3924 privilegesDropToUid = to!uid_t(arg); 3925 foundUid = false; 3926 continue; 3927 } 3928 if(foundGid) { 3929 privilegesDropToGid = to!gid_t(arg); 3930 foundGid = false; 3931 continue; 3932 } 3933 if(foundListen) { 3934 this.listenSpec ~= arg; 3935 foundListen = false; 3936 continue; 3937 } 3938 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3939 foundHost = true; 3940 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3941 foundPort = true; 3942 else if(arg == "--uid") 3943 foundUid = true; 3944 else if(arg == "--gid") 3945 foundGid = true; 3946 else if(arg == "--listen") 3947 foundListen = true; 3948 } 3949 3950 if(portOrHostFound && listenSpec.length) { 3951 throw new Exception("You passed both a --listening-host or --listening-port and a --listen argument. You should fix your script to ONLY use --listen arguments."); 3952 } 3953 } 3954 3955 version(Windows) { 3956 private alias uid_t = int; 3957 private alias gid_t = int; 3958 } 3959 3960 /// user (uid) to drop privileges to 3961 /// 0 … do nothing 3962 uid_t privilegesDropToUid = 0; 3963 /// group (gid) to drop privileges to 3964 /// 0 … do nothing 3965 gid_t privilegesDropToGid = 0; 3966 3967 private void dropPrivileges() { 3968 version(Posix) { 3969 import core.sys.posix.unistd; 3970 3971 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 3972 throw new Exception("Dropping privileges via setgid() failed."); 3973 3974 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 3975 throw new Exception("Dropping privileges via setuid() failed."); 3976 } 3977 else { 3978 // FIXME: Windows? 3979 //pragma(msg, "Dropping privileges is not implemented for this platform"); 3980 } 3981 3982 // done, set zero 3983 privilegesDropToGid = 0; 3984 privilegesDropToUid = 0; 3985 } 3986 3987 /++ 3988 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 3989 3990 History: 3991 Added Oct 10, 2020. 3992 Example: 3993 3994 --- 3995 import arsd.cgi; 3996 void main() { 3997 RequestServer server = RequestServer("127.0.0.1", 6789); 3998 string oauthCode; 3999 string oauthScope; 4000 server.serveHttpOnce!((cgi) { 4001 oauthCode = cgi.request("code"); 4002 oauthScope = cgi.request("scope"); 4003 cgi.write("Thank you, please return to the application."); 4004 }); 4005 // use the code and scope given 4006 } 4007 --- 4008 +/ 4009 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4010 import std.socket; 4011 4012 bool tcp; 4013 void delegate() cleanup; 4014 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 4015 auto connection = socket.accept(); 4016 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 4017 4018 if(cleanup) 4019 cleanup(); 4020 } 4021 4022 /++ 4023 Starts serving requests according to the current configuration. 4024 +/ 4025 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4026 version(netman_httpd) { 4027 // Obsolete! 4028 4029 import arsd.httpd; 4030 // what about forwarding the other constructor args? 4031 // this probably needs a whole redoing... 4032 serveHttp!CustomCgi(&fun, listeningPort);//5005); 4033 return; 4034 } else 4035 version(embedded_httpd_processes) { 4036 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 4037 } else 4038 version(embedded_httpd_threads) { 4039 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 4040 } else 4041 version(scgi) { 4042 serveScgi!(fun, CustomCgi, maxContentLength)(); 4043 } else 4044 version(fastcgi) { 4045 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 4046 } else 4047 version(stdio_http) { 4048 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 4049 } else { 4050 //version=plain_cgi; 4051 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 4052 } 4053 } 4054 4055 /++ 4056 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 4057 4058 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 4059 +/ 4060 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 4061 globalStopFlag = false; 4062 static if(__traits(isStaticFunction, fun)) 4063 alias funToUse = fun; 4064 else 4065 void funToUse(CustomCgi cgi) { 4066 static if(__VERSION__ > 2097) 4067 __traits(child, _this, fun)(cgi); 4068 else static assert(0, "Not implemented in your compiler version!"); 4069 } 4070 auto manager = this.listenSpec is null ? 4071 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads) : 4072 new ListeningConnectionManager(this.listenSpec, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 4073 manager.listen(); 4074 } 4075 4076 /++ 4077 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 4078 +/ 4079 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4080 globalStopFlag = false; 4081 auto manager = this.listenSpec is null ? 4082 new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads) : 4083 new ListeningConnectionManager(this.listenSpec, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 4084 manager.listen(); 4085 } 4086 4087 /++ 4088 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 4089 4090 Intended for cases like working from systemd, like discussed here: [https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org] 4091 4092 History: 4093 Added May 29, 2021 4094 +/ 4095 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4096 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 4097 } 4098 4099 /++ 4100 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 4101 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 4102 +/ 4103 enum ForceStop { 4104 /++ 4105 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 4106 +/ 4107 afterQueuedRequestsComplete, 4108 /++ 4109 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 4110 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 4111 +/ 4112 afterCurrentRequestsComplete, 4113 /++ 4114 Partial response writes will throw an exception, cancelling any streaming response, but complete 4115 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 4116 +/ 4117 cancelStreamingRequestsEarly, 4118 /++ 4119 All writes will throw. 4120 +/ 4121 cancelAllRequestsEarly, 4122 /++ 4123 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 4124 +/ 4125 forciblyTerminate, 4126 } 4127 4128 version(embedded_httpd_processes) {} else 4129 /++ 4130 Stops serving after the current requests are completed. 4131 4132 Bugs: 4133 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 4134 on Windows however). Only partially implemented on non-Linux posix systems. 4135 4136 You might also try SIGINT perhaps. 4137 4138 The stopPriority is not yet fully implemented. 4139 +/ 4140 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 4141 globalStopFlag = true; 4142 4143 version(Posix) { 4144 if(cancelfd > 0) { 4145 ulong a = 1; 4146 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 4147 } 4148 } 4149 version(Windows) { 4150 if(iocp) { 4151 foreach(i; 0 .. 16) // FIXME 4152 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 4153 } 4154 } 4155 } 4156 } 4157 4158 private alias AliasSeq(T...) = T; 4159 4160 version(with_breaking_cgi_features) 4161 mixin(q{ 4162 template ThisFor(alias t) { 4163 static if(__traits(isStaticFunction, t)) { 4164 alias ThisFor = AliasSeq!(); 4165 } else { 4166 alias ThisFor = __traits(parent, t); 4167 } 4168 } 4169 }); 4170 else 4171 alias ThisFor(alias t) = AliasSeq!(); 4172 4173 private __gshared bool globalStopFlag = false; 4174 4175 version(embedded_httpd_processes) 4176 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 4177 import core.sys.posix.unistd; 4178 import core.sys.posix.sys.socket; 4179 import core.sys.posix.netinet.in_; 4180 //import std.c.linux.socket; 4181 4182 int sock = socket(AF_INET, SOCK_STREAM, 0); 4183 if(sock == -1) 4184 throw new Exception("socket"); 4185 4186 cloexec(sock); 4187 4188 { 4189 4190 sockaddr_in addr; 4191 addr.sin_family = AF_INET; 4192 addr.sin_port = htons(params.listeningPort); 4193 auto lh = params.listeningHost; 4194 if(lh.length) { 4195 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 4196 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 4197 } else 4198 addr.sin_addr.s_addr = INADDR_ANY; 4199 4200 // HACKISH 4201 int on = 1; 4202 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 4203 // end hack 4204 4205 4206 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 4207 close(sock); 4208 throw new Exception("bind"); 4209 } 4210 4211 // FIXME: if this queue is full, it will just ignore it 4212 // and wait for the client to retransmit it. This is an 4213 // obnoxious timeout condition there. 4214 if(sock.listen(128) == -1) { 4215 close(sock); 4216 throw new Exception("listen"); 4217 } 4218 params.dropPrivileges(); 4219 } 4220 4221 version(embedded_httpd_processes_accept_after_fork) {} else { 4222 int pipeReadFd; 4223 int pipeWriteFd; 4224 4225 { 4226 int[2] pipeFd; 4227 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 4228 import core.stdc.errno; 4229 throw new Exception("pipe failed " ~ to!string(errno)); 4230 } 4231 4232 pipeReadFd = pipeFd[0]; 4233 pipeWriteFd = pipeFd[1]; 4234 } 4235 } 4236 4237 4238 int processCount; 4239 pid_t newPid; 4240 reopen: 4241 while(processCount < processPoolSize) { 4242 newPid = fork(); 4243 if(newPid == 0) { 4244 // start serving on the socket 4245 //ubyte[4096] backingBuffer; 4246 for(;;) { 4247 bool closeConnection; 4248 uint i; 4249 sockaddr addr; 4250 i = addr.sizeof; 4251 version(embedded_httpd_processes_accept_after_fork) { 4252 int s = accept(sock, &addr, &i); 4253 int opt = 1; 4254 import core.sys.posix.netinet.tcp; 4255 // the Cgi class does internal buffering, so disabling this 4256 // helps with latency in many cases... 4257 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4258 cloexec(s); 4259 } else { 4260 int s; 4261 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 4262 if(readret != s.sizeof) { 4263 import core.stdc.errno; 4264 throw new Exception("pipe read failed " ~ to!string(errno)); 4265 } 4266 4267 //writeln("process ", getpid(), " got socket ", s); 4268 } 4269 4270 try { 4271 4272 if(s == -1) 4273 throw new Exception("accept"); 4274 4275 scope(failure) close(s); 4276 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 4277 auto ir = new BufferedInputRange(s); 4278 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 4279 4280 while(!ir.empty) { 4281 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 4282 4283 Cgi cgi; 4284 try { 4285 cgi = new CustomCgi(ir, &closeConnection); 4286 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 4287 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 4288 if(processPoolSize <= 1) 4289 closeConnection = true; 4290 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 4291 } catch(HttpVersionNotSupportedException he) { 4292 sendAll(ir.source, plainHttpError(false, "505 HTTP Version Not Supported", he)); 4293 closeConnection = true; 4294 break; 4295 } catch(Throwable t) { 4296 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4297 // anyway let's kill the connection 4298 version(CRuntime_Musl) { 4299 // LockingTextWriter fails here 4300 // so working around it 4301 auto estr = t.toString(); 4302 stderr.rawWrite(estr); 4303 stderr.rawWrite("\n"); 4304 } else 4305 stderr.writeln(t.toString()); 4306 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 4307 closeConnection = true; 4308 break; 4309 } 4310 assert(cgi !is null); 4311 scope(exit) 4312 cgi.dispose(); 4313 4314 try { 4315 fun(cgi); 4316 cgi.close(); 4317 if(cgi.websocketMode) 4318 closeConnection = true; 4319 } catch(ConnectionException ce) { 4320 closeConnection = true; 4321 } catch(Throwable t) { 4322 // a processing error can be recovered from 4323 version(CRuntime_Musl) { 4324 // LockingTextWriter fails here 4325 // so working around it 4326 auto estr = t.toString(); 4327 stderr.rawWrite(estr); 4328 } else { 4329 stderr.writeln(t.toString); 4330 } 4331 if(!handleException(cgi, t)) 4332 closeConnection = true; 4333 } 4334 4335 if(closeConnection) { 4336 ir.source.close(); 4337 break; 4338 } else { 4339 if(!ir.empty) 4340 ir.popFront(); // get the next 4341 else if(ir.sourceClosed) { 4342 ir.source.close(); 4343 } 4344 } 4345 } 4346 4347 ir.source.close(); 4348 } catch(Throwable t) { 4349 version(CRuntime_Musl) {} else 4350 debug writeln(t); 4351 // most likely cause is a timeout 4352 } 4353 } 4354 } else if(newPid < 0) { 4355 throw new Exception("fork failed"); 4356 } else { 4357 processCount++; 4358 } 4359 } 4360 4361 // the parent should wait for its children... 4362 if(newPid) { 4363 import core.sys.posix.sys.wait; 4364 4365 version(embedded_httpd_processes_accept_after_fork) {} else { 4366 import core.sys.posix.sys.select; 4367 int[] fdQueue; 4368 while(true) { 4369 // writeln("select call"); 4370 int nfds = pipeWriteFd; 4371 if(sock > pipeWriteFd) 4372 nfds = sock; 4373 nfds += 1; 4374 fd_set read_fds; 4375 fd_set write_fds; 4376 FD_ZERO(&read_fds); 4377 FD_ZERO(&write_fds); 4378 FD_SET(sock, &read_fds); 4379 if(fdQueue.length) 4380 FD_SET(pipeWriteFd, &write_fds); 4381 auto ret = select(nfds, &read_fds, &write_fds, null, null); 4382 if(ret == -1) { 4383 import core.stdc.errno; 4384 if(errno == EINTR) 4385 goto try_wait; 4386 else 4387 throw new Exception("wtf select"); 4388 } 4389 4390 int s = -1; 4391 if(FD_ISSET(sock, &read_fds)) { 4392 uint i; 4393 sockaddr addr; 4394 i = addr.sizeof; 4395 s = accept(sock, &addr, &i); 4396 cloexec(s); 4397 import core.sys.posix.netinet.tcp; 4398 int opt = 1; 4399 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4400 } 4401 4402 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4403 if(s == -1 && fdQueue.length) { 4404 s = fdQueue[0]; 4405 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4406 } 4407 write_fd(pipeWriteFd, &s, s.sizeof, s); 4408 close(s); // we are done with it, let the other process take ownership 4409 } else 4410 fdQueue ~= s; 4411 } 4412 } 4413 4414 try_wait: 4415 4416 int status; 4417 while(-1 != wait(&status)) { 4418 version(CRuntime_Musl) {} else { 4419 import std.stdio; writeln("Process died ", status); 4420 } 4421 processCount--; 4422 goto reopen; 4423 } 4424 close(sock); 4425 } 4426 } 4427 4428 version(fastcgi) 4429 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4430 // SetHandler fcgid-script 4431 FCGX_Stream* input, output, error; 4432 FCGX_ParamArray env; 4433 4434 4435 4436 const(ubyte)[] getFcgiChunk() { 4437 const(ubyte)[] ret; 4438 while(FCGX_HasSeenEOF(input) != -1) 4439 ret ~= cast(ubyte) FCGX_GetChar(input); 4440 return ret; 4441 } 4442 4443 void writeFcgi(const(ubyte)[] data) { 4444 FCGX_PutStr(data.ptr, data.length, output); 4445 } 4446 4447 void doARequest() { 4448 string[string] fcgienv; 4449 4450 for(auto e = env; e !is null && *e !is null; e++) { 4451 string cur = to!string(*e); 4452 auto idx = cur.indexOf("="); 4453 string name, value; 4454 if(idx == -1) 4455 name = cur; 4456 else { 4457 name = cur[0 .. idx]; 4458 value = cur[idx + 1 .. $]; 4459 } 4460 4461 fcgienv[name] = value; 4462 } 4463 4464 void flushFcgi() { 4465 FCGX_FFlush(output); 4466 } 4467 4468 Cgi cgi; 4469 try { 4470 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4471 } catch(Throwable t) { 4472 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4473 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4474 return; //continue; 4475 } 4476 assert(cgi !is null); 4477 scope(exit) cgi.dispose(); 4478 try { 4479 fun(cgi); 4480 cgi.close(); 4481 } catch(Throwable t) { 4482 // log it to the error stream 4483 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4484 // handle it for the user, if we can 4485 if(!handleException(cgi, t)) 4486 return; // continue; 4487 } 4488 } 4489 4490 auto lp = params.listeningPort; 4491 auto host = params.listeningHost; 4492 4493 FCGX_Request request; 4494 if(lp || !host.empty) { 4495 // if a listening port was specified on the command line, we want to spawn ourself 4496 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4497 FCGX_Init(); 4498 4499 int sock; 4500 4501 if(host.startsWith("unix:")) { 4502 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4503 } else if(host.startsWith("abstract:")) { 4504 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4505 } else { 4506 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4507 } 4508 4509 if(sock < 0) 4510 throw new Exception("Couldn't listen on the port"); 4511 FCGX_InitRequest(&request, sock, 0); 4512 while(FCGX_Accept_r(&request) >= 0) { 4513 input = request.inStream; 4514 output = request.outStream; 4515 error = request.errStream; 4516 env = request.envp; 4517 doARequest(); 4518 } 4519 } else { 4520 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4521 // using the version with a global variable since we are separate processes anyway 4522 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4523 doARequest(); 4524 } 4525 } 4526 } 4527 4528 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4529 ushort defaultListeningPort() { 4530 version(netman_httpd) 4531 return 8080; 4532 else version(embedded_httpd_processes) 4533 return 8085; 4534 else version(embedded_httpd_threads) 4535 return 8085; 4536 else version(scgi) 4537 return 4000; 4538 else 4539 return 0; 4540 } 4541 4542 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4543 string defaultListeningHost() { 4544 version(netman_httpd) 4545 return null; 4546 else version(embedded_httpd_processes) 4547 return null; 4548 else version(embedded_httpd_threads) 4549 return null; 4550 else version(scgi) 4551 return "127.0.0.1"; 4552 else 4553 return null; 4554 4555 } 4556 4557 /++ 4558 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4559 4560 Please note that this may spawn other helper processes that will call `main` again. It does this currently for the timer server and event source server (and the quasi-deprecated web socket server). 4561 4562 Params: 4563 fun = Your request handler 4564 CustomCgi = a subclass of Cgi, if you wise to customize it further 4565 maxContentLength = max POST size you want to allow 4566 args = command-line arguments 4567 4568 History: 4569 Documented Sept 26, 2020. 4570 +/ 4571 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4572 if(tryAddonServers(args)) 4573 return; 4574 4575 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4576 return; 4577 4578 RequestServer server; 4579 // you can change the port here if you like 4580 // server.listeningPort = 9000; 4581 4582 // then call this to let the command line args override your default 4583 server.configureFromCommandLine(args); 4584 4585 // and serve the request(s). 4586 server.serve!(fun, CustomCgi, maxContentLength)(); 4587 } 4588 4589 //version(plain_cgi) 4590 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4591 // standard CGI is the default version 4592 4593 4594 // Set stdin to binary mode if necessary to avoid mangled newlines 4595 // the fact that stdin is global means this could be trouble but standard cgi request 4596 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4597 version(Windows) { 4598 version(Win64) 4599 _setmode(std.stdio.stdin.fileno(), 0x8000); 4600 else 4601 setmode(std.stdio.stdin.fileno(), 0x8000); 4602 } 4603 4604 Cgi cgi; 4605 try { 4606 cgi = new CustomCgi(maxContentLength); 4607 version(Posix) 4608 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4609 else version(Windows) 4610 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4611 else static assert(0); 4612 } catch(Throwable t) { 4613 version(CRuntime_Musl) { 4614 // LockingTextWriter fails here 4615 // so working around it 4616 auto s = t.toString(); 4617 stderr.rawWrite(s); 4618 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4619 } else { 4620 stderr.writeln(t.msg); 4621 // the real http server will probably handle this; 4622 // most likely, this is a bug in Cgi. But, oh well. 4623 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4624 } 4625 return; 4626 } 4627 assert(cgi !is null); 4628 scope(exit) cgi.dispose(); 4629 4630 try { 4631 fun(cgi); 4632 cgi.close(); 4633 } catch (Throwable t) { 4634 version(CRuntime_Musl) { 4635 // LockingTextWriter fails here 4636 // so working around it 4637 auto s = t.msg; 4638 stderr.rawWrite(s); 4639 } else { 4640 stderr.writeln(t.msg); 4641 } 4642 if(!handleException(cgi, t)) 4643 return; 4644 } 4645 } 4646 4647 private __gshared int cancelfd = -1; 4648 4649 /+ 4650 The event loop for embedded_httpd_threads will prolly fiber dispatch 4651 cgi constructors too, so slow posts will not monopolize a worker thread. 4652 4653 May want to provide the worker task system just need to ensure all the fibers 4654 has a big enough stack for real work... would also ideally like to reuse them. 4655 4656 4657 So prolly bir would switch it to nonblocking. If it would block, it epoll 4658 registers one shot with this existing fiber to take it over. 4659 4660 new connection comes in. it picks a fiber off the free list, 4661 or if there is none, it creates a new one. this fiber handles 4662 this connection the whole time. 4663 4664 epoll triggers the fiber when something comes in. it is called by 4665 a random worker thread, it might change at any time. at least during 4666 the constructor. maybe into the main body it will stay tied to a thread 4667 just so TLS stuff doesn't randomly change in the middle. but I could 4668 specify if you yield all bets are off. 4669 4670 when the request is finished, if there's more data buffered, it just 4671 keeps going. if there is no more data buffered, it epoll ctls to 4672 get triggered when more data comes in. all one shot. 4673 4674 when a connection is closed, the fiber returns and is then reset 4675 and added to the free list. if the free list is full, the fiber is 4676 just freed, this means it will balloon to a certain size but not generally 4677 grow beyond that unless the activity keeps going. 4678 4679 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4680 4681 So the fiber has its own magic methods to read and write. if they would block, it registers 4682 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4683 4684 basically you issue the command and it tells you when it is done 4685 4686 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4687 4688 +/ 4689 4690 /++ 4691 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4692 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4693 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4694 4695 History: 4696 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4697 +/ 4698 version(cgi_use_fiber) 4699 __gshared size_t fiberStackSize = 4096 * 100; 4700 4701 version(cgi_use_fiber) 4702 class CgiFiber : Fiber { 4703 private void function(Socket) f_handler; 4704 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4705 f_handler(s); 4706 } 4707 this(void function(Socket) handler) { 4708 this.f_handler = handler; 4709 this(&f_handler_dg); 4710 } 4711 4712 this(void delegate(Socket) handler) { 4713 this.handler = handler; 4714 super(&run, fiberStackSize); 4715 } 4716 4717 Socket connection; 4718 void delegate(Socket) handler; 4719 4720 void run() { 4721 handler(connection); 4722 } 4723 4724 void delegate() postYield; 4725 4726 private void setPostYield(scope void delegate() py) @nogc { 4727 postYield = cast(void delegate()) py; 4728 } 4729 4730 void proceed() { 4731 try { 4732 call(); 4733 auto py = postYield; 4734 postYield = null; 4735 if(py !is null) 4736 py(); 4737 } catch(Exception e) { 4738 if(connection) 4739 connection.close(); 4740 goto terminate; 4741 } 4742 4743 if(state == State.TERM) { 4744 terminate: 4745 import core.memory; 4746 GC.removeRoot(cast(void*) this); 4747 } 4748 } 4749 } 4750 4751 version(cgi_use_fiber) 4752 version(Windows) { 4753 4754 extern(Windows) private { 4755 4756 import core.sys.windows.mswsock; 4757 4758 alias GROUP=uint; 4759 alias LPWSAPROTOCOL_INFOW = void*; 4760 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4761 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4762 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4763 4764 struct WSABUF { 4765 ULONG len; 4766 CHAR *buf; 4767 } 4768 alias LPWSABUF = WSABUF*; 4769 4770 alias WSAOVERLAPPED = OVERLAPPED; 4771 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4772 /+ 4773 4774 alias LPFN_ACCEPTEX = 4775 BOOL 4776 function( 4777 SOCKET sListenSocket, 4778 SOCKET sAcceptSocket, 4779 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4780 void* lpOutputBuffer, 4781 WORD dwReceiveDataLength, 4782 WORD dwLocalAddressLength, 4783 WORD dwRemoteAddressLength, 4784 LPDWORD lpdwBytesReceived, 4785 LPOVERLAPPED lpOverlapped 4786 ); 4787 4788 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4789 +/ 4790 4791 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4792 } 4793 4794 private class PseudoblockingOverlappedSocket : Socket { 4795 SOCKET handle; 4796 4797 CgiFiber fiber; 4798 4799 this(AddressFamily af, SocketType st) { 4800 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4801 if(!handle) 4802 throw new Exception("WSASocketW"); 4803 this.handle = handle; 4804 4805 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4806 4807 if(iocp is null) { 4808 writeln(GetLastError()); 4809 throw new Exception("CreateIoCompletionPort"); 4810 } 4811 4812 super(cast(socket_t) handle, af); 4813 } 4814 this() pure nothrow @trusted { assert(0); } 4815 4816 override void blocking(bool) {} // meaningless to us, just ignore it. 4817 4818 protected override Socket accepting() pure nothrow { 4819 assert(0); 4820 } 4821 4822 bool addressesParsed; 4823 Address la; 4824 Address ra; 4825 4826 private void populateAddresses() { 4827 if(addressesParsed) 4828 return; 4829 addressesParsed = true; 4830 4831 int lalen, ralen; 4832 4833 sockaddr_in* la; 4834 sockaddr_in* ra; 4835 4836 lpfnGetAcceptExSockaddrs( 4837 scratchBuffer.ptr, 4838 0, // same as in the AcceptEx call! 4839 sockaddr_in.sizeof + 16, 4840 sockaddr_in.sizeof + 16, 4841 cast(sockaddr**) &la, 4842 &lalen, 4843 cast(sockaddr**) &ra, 4844 &ralen 4845 ); 4846 4847 if(la) 4848 this.la = new InternetAddress(*la); 4849 if(ra) 4850 this.ra = new InternetAddress(*ra); 4851 4852 } 4853 4854 override @property @trusted Address localAddress() { 4855 populateAddresses(); 4856 return la; 4857 } 4858 override @property @trusted Address remoteAddress() { 4859 populateAddresses(); 4860 return ra; 4861 } 4862 4863 PseudoblockingOverlappedSocket accepted; 4864 4865 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4866 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4867 4868 override Socket accept() @trusted { 4869 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4870 4871 if(lpfnAcceptEx is null) { 4872 DWORD dwBytes; 4873 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4874 4875 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4876 &GuidAcceptEx, GuidAcceptEx.sizeof, 4877 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4878 &dwBytes, null, null); 4879 4880 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4881 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4882 &GuidAcceptEx, GuidAcceptEx.sizeof, 4883 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4884 &dwBytes, null, null); 4885 4886 } 4887 4888 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4889 accepted = pfa; 4890 4891 SOCKET pendingForAccept = pfa.handle; 4892 DWORD ignored; 4893 4894 auto ret = lpfnAcceptEx(handle, 4895 pendingForAccept, 4896 // buffer to receive up front 4897 pfa.scratchBuffer.ptr, 4898 0, 4899 // size of local and remote addresses. normally + 16. 4900 sockaddr_in.sizeof + 16, 4901 sockaddr_in.sizeof + 16, 4902 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4903 &overlapped 4904 ); 4905 4906 return pfa; 4907 } 4908 4909 override void connect(Address to) { assert(0); } 4910 4911 DWORD lastAnswer; 4912 ubyte[1024] scratchBuffer; 4913 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4914 4915 WSABUF[1] buffer; 4916 OVERLAPPED overlapped; 4917 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4918 overlapped = overlapped.init; 4919 buffer[0].len = cast(DWORD) buf.length; 4920 buffer[0].buf = cast(CHAR*) buf.ptr; 4921 fiber.setPostYield( () { 4922 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4923 if(GetLastError() != 997) { 4924 //throw new Exception("WSASend fail"); 4925 } 4926 } 4927 }); 4928 4929 Fiber.yield(); 4930 return lastAnswer; 4931 } 4932 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 4933 overlapped = overlapped.init; 4934 buffer[0].len = cast(DWORD) buf.length; 4935 buffer[0].buf = cast(CHAR*) buf.ptr; 4936 4937 DWORD flags2 = 0; 4938 4939 fiber.setPostYield(() { 4940 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 4941 if(GetLastError() != 997) { 4942 //writeln("WSARecv ", WSAGetLastError()); 4943 //throw new Exception("WSARecv fail"); 4944 } 4945 } 4946 }); 4947 4948 Fiber.yield(); 4949 return lastAnswer; 4950 } 4951 4952 // I might go back and implement these for udp things. 4953 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 4954 assert(0); 4955 } 4956 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 4957 assert(0); 4958 } 4959 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 4960 assert(0); 4961 } 4962 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 4963 assert(0); 4964 } 4965 4966 // lol overload sets 4967 alias send = typeof(super).send; 4968 alias receive = typeof(super).receive; 4969 alias sendTo = typeof(super).sendTo; 4970 alias receiveFrom = typeof(super).receiveFrom; 4971 4972 } 4973 } 4974 4975 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 4976 assert(connection !is null); 4977 version(cgi_use_fiber) { 4978 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 4979 4980 version(Windows) { 4981 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 4982 } 4983 4984 import core.memory; 4985 GC.addRoot(cast(void*) fiber); 4986 fiber.connection = connection; 4987 fiber.proceed(); 4988 } else { 4989 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 4990 } 4991 } 4992 4993 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 4994 scope(failure) { 4995 // catch all for other errors 4996 try { 4997 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 4998 connection.close(); 4999 } catch(Exception e) {} // swallow it, we're aborting anyway. 5000 } 5001 5002 bool closeConnection = alwaysCloseConnection; 5003 5004 /+ 5005 ubyte[4096] inputBuffer = void; 5006 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 5007 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 5008 5009 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 5010 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 5011 ir.__ctor(connection, inputBuffer[], true); 5012 +/ 5013 5014 auto ir = new BufferedInputRange(connection); 5015 5016 while(!ir.empty) { 5017 5018 if(ir.view.length == 0) { 5019 ir.popFront(); 5020 if(ir.sourceClosed) { 5021 connection.close(); 5022 closeConnection = true; 5023 break; 5024 } 5025 } 5026 5027 Cgi cgi; 5028 try { 5029 cgi = new CustomCgi(ir, &closeConnection); 5030 // There's a bunch of these casts around because the type matches up with 5031 // the -version=.... specifiers, just you can also create a RequestServer 5032 // and instantiate the things where the types don't match up. It isn't exactly 5033 // correct but I also don't care rn. Might FIXME and either remove it later or something. 5034 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5035 } catch(ConnectionClosedException ce) { 5036 closeConnection = true; 5037 break; 5038 } catch(ConnectionException ce) { 5039 // broken pipe or something, just abort the connection 5040 closeConnection = true; 5041 break; 5042 } catch(HttpVersionNotSupportedException ve) { 5043 sendAll(connection, plainHttpError(false, "505 HTTP Version Not Supported", ve)); 5044 closeConnection = true; 5045 break; 5046 } catch(Throwable t) { 5047 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 5048 // anyway let's kill the connection 5049 version(CRuntime_Musl) { 5050 stderr.rawWrite(t.toString()); 5051 stderr.rawWrite("\n"); 5052 } else { 5053 stderr.writeln(t.toString()); 5054 } 5055 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 5056 closeConnection = true; 5057 break; 5058 } 5059 assert(cgi !is null); 5060 scope(exit) 5061 cgi.dispose(); 5062 5063 try { 5064 fun(cgi); 5065 cgi.close(); 5066 if(cgi.websocketMode) 5067 closeConnection = true; 5068 } catch(ConnectionException ce) { 5069 // broken pipe or something, just abort the connection 5070 closeConnection = true; 5071 } catch(ConnectionClosedException ce) { 5072 // broken pipe or something, just abort the connection 5073 closeConnection = true; 5074 } catch(Throwable t) { 5075 // a processing error can be recovered from 5076 version(CRuntime_Musl) {} else 5077 stderr.writeln(t.toString); 5078 if(!handleException(cgi, t)) 5079 closeConnection = true; 5080 } 5081 5082 if(globalStopFlag) 5083 closeConnection = true; 5084 5085 if(closeConnection || alwaysCloseConnection) { 5086 connection.shutdown(SocketShutdown.BOTH); 5087 connection.close(); 5088 ir.dispose(); 5089 closeConnection = false; // don't reclose after loop 5090 break; 5091 } else { 5092 if(ir.front.length) { 5093 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 5094 } else if(ir.sourceClosed) { 5095 ir.source.shutdown(SocketShutdown.BOTH); 5096 ir.source.close(); 5097 ir.dispose(); 5098 closeConnection = false; 5099 } else { 5100 continue; 5101 // break; // this was for a keepalive experiment 5102 } 5103 } 5104 } 5105 5106 if(closeConnection) { 5107 connection.shutdown(SocketShutdown.BOTH); 5108 connection.close(); 5109 ir.dispose(); 5110 } 5111 5112 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 5113 } 5114 5115 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 5116 // and now we can buffer 5117 scope(failure) 5118 connection.close(); 5119 5120 import al = std.algorithm; 5121 5122 size_t size; 5123 5124 string[string] headers; 5125 5126 auto range = new BufferedInputRange(connection); 5127 more_data: 5128 auto chunk = range.front(); 5129 // waiting for colon for header length 5130 auto idx = indexOf(cast(string) chunk, ':'); 5131 if(idx == -1) { 5132 try { 5133 range.popFront(); 5134 } catch(Exception e) { 5135 // it is just closed, no big deal 5136 connection.close(); 5137 return; 5138 } 5139 goto more_data; 5140 } 5141 5142 size = to!size_t(cast(string) chunk[0 .. idx]); 5143 chunk = range.consume(idx + 1); 5144 // reading headers 5145 if(chunk.length < size) 5146 range.popFront(0, size + 1); 5147 // we are now guaranteed to have enough 5148 chunk = range.front(); 5149 assert(chunk.length > size); 5150 5151 idx = 0; 5152 string key; 5153 string value; 5154 foreach(part; al.splitter(chunk, '\0')) { 5155 if(idx & 1) { // odd is value 5156 value = cast(string)(part.idup); 5157 headers[key] = value; // commit 5158 } else 5159 key = cast(string)(part.idup); 5160 idx++; 5161 } 5162 5163 enforce(chunk[size] == ','); // the terminator 5164 5165 range.consume(size + 1); 5166 // reading data 5167 // this will be done by Cgi 5168 5169 const(ubyte)[] getScgiChunk() { 5170 // we are already primed 5171 auto data = range.front(); 5172 if(data.length == 0 && !range.sourceClosed) { 5173 range.popFront(0); 5174 data = range.front(); 5175 } else if (range.sourceClosed) 5176 range.source.close(); 5177 5178 return data; 5179 } 5180 5181 void writeScgi(const(ubyte)[] data) { 5182 sendAll(connection, data); 5183 } 5184 5185 void flushScgi() { 5186 // I don't *think* I have to do anything.... 5187 } 5188 5189 Cgi cgi; 5190 try { 5191 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 5192 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 5193 } catch(Throwable t) { 5194 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 5195 connection.close(); 5196 return; // this connection is dead 5197 } 5198 assert(cgi !is null); 5199 scope(exit) cgi.dispose(); 5200 try { 5201 fun(cgi); 5202 cgi.close(); 5203 connection.close(); 5204 } catch(Throwable t) { 5205 // no std err 5206 if(!handleException(cgi, t)) { 5207 connection.close(); 5208 return; 5209 } else { 5210 connection.close(); 5211 return; 5212 } 5213 } 5214 } 5215 5216 string printDate(DateTime date) { 5217 char[29] buffer = void; 5218 printDateToBuffer(date, buffer[]); 5219 return buffer.idup; 5220 } 5221 5222 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 5223 assert(buffer.length >= 29); 5224 // 29 static length ? 5225 5226 static immutable daysOfWeek = [ 5227 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 5228 ]; 5229 5230 static immutable months = [ 5231 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 5232 ]; 5233 5234 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 5235 buffer[3 .. 5] = ", "; 5236 buffer[5] = date.day / 10 + '0'; 5237 buffer[6] = date.day % 10 + '0'; 5238 buffer[7] = ' '; 5239 buffer[8 .. 11] = months[date.month]; 5240 buffer[11] = ' '; 5241 auto y = date.year; 5242 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 5243 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 5244 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 5245 buffer[15] = cast(char) (y + '0'); 5246 buffer[16] = ' '; 5247 buffer[17] = date.hour / 10 + '0'; 5248 buffer[18] = date.hour % 10 + '0'; 5249 buffer[19] = ':'; 5250 buffer[20] = date.minute / 10 + '0'; 5251 buffer[21] = date.minute % 10 + '0'; 5252 buffer[22] = ':'; 5253 buffer[23] = date.second / 10 + '0'; 5254 buffer[24] = date.second % 10 + '0'; 5255 buffer[25 .. $] = " GMT"; 5256 5257 return 29; 5258 } 5259 5260 5261 // Referencing this gigantic typeid seems to remind the compiler 5262 // to actually put the symbol in the object file. I guess the immutable 5263 // assoc array array isn't actually included in druntime 5264 void hackAroundLinkerError() { 5265 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 5266 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 5267 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 5268 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 5269 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 5270 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 5271 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 5272 // this is getting kinda ridiculous btw. Moving assoc arrays 5273 // to the library is the pain that keeps on coming. 5274 5275 // eh this broke the build on the work server 5276 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 5277 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 5278 } 5279 5280 5281 5282 5283 5284 version(fastcgi) { 5285 pragma(lib, "fcgi"); 5286 5287 static if(size_t.sizeof == 8) // 64 bit 5288 alias long c_int; 5289 else 5290 alias int c_int; 5291 5292 extern(C) { 5293 struct FCGX_Stream { 5294 ubyte* rdNext; 5295 ubyte* wrNext; 5296 ubyte* stop; 5297 ubyte* stopUnget; 5298 c_int isReader; 5299 c_int isClosed; 5300 c_int wasFCloseCalled; 5301 c_int FCGI_errno; 5302 void* function(FCGX_Stream* stream) fillBuffProc; 5303 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 5304 void* data; 5305 } 5306 5307 // note: this is meant to be opaque, so don't access it directly 5308 struct FCGX_Request { 5309 int requestId; 5310 int role; 5311 FCGX_Stream* inStream; 5312 FCGX_Stream* outStream; 5313 FCGX_Stream* errStream; 5314 char** envp; 5315 void* paramsPtr; 5316 int ipcFd; 5317 int isBeginProcessed; 5318 int keepConnection; 5319 int appStatus; 5320 int nWriters; 5321 int flags; 5322 int listen_sock; 5323 } 5324 5325 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 5326 void FCGX_Init(); 5327 5328 int FCGX_Accept_r(FCGX_Request *request); 5329 5330 5331 alias char** FCGX_ParamArray; 5332 5333 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 5334 c_int FCGX_GetChar(FCGX_Stream* stream); 5335 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 5336 int FCGX_HasSeenEOF(FCGX_Stream* stream); 5337 c_int FCGX_FFlush(FCGX_Stream *stream); 5338 5339 int FCGX_OpenSocket(in char*, int); 5340 } 5341 } 5342 5343 5344 /* This might go int a separate module eventually. It is a network input helper class. */ 5345 5346 import std.socket; 5347 5348 version(cgi_use_fiber) { 5349 import core.thread; 5350 5351 version(linux) { 5352 import core.sys.linux.epoll; 5353 5354 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 5355 } else version(Windows) { 5356 // declaring the iocp thing below... 5357 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 5358 } 5359 5360 version(Windows) 5361 __gshared HANDLE iocp; 5362 5363 version(cgi_use_fiber) { 5364 version(linux) 5365 private enum WakeupEvent { 5366 Read = EPOLLIN, 5367 Write = EPOLLOUT 5368 } 5369 else version(Windows) 5370 private enum WakeupEvent { 5371 Read, Write 5372 } 5373 else static assert(0); 5374 } 5375 5376 version(cgi_use_fiber) 5377 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 5378 5379 // static cast since I know what i have in here and don't want to pay for dynamic cast 5380 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 5381 5382 version(linux) { 5383 f.setPostYield = () { 5384 if(*registered) { 5385 // rearm 5386 epoll_event evt; 5387 evt.events = e | EPOLLONESHOT; 5388 evt.data.ptr = cast(void*) f; 5389 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 5390 throw new Exception("epoll_ctl"); 5391 } else { 5392 // initial registration 5393 *registered = true ; 5394 int fd = source.handle; 5395 epoll_event evt; 5396 evt.events = e | EPOLLONESHOT; 5397 evt.data.ptr = cast(void*) f; 5398 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5399 throw new Exception("epoll_ctl"); 5400 } 5401 }; 5402 5403 Fiber.yield(); 5404 5405 f.setPostYield(null); 5406 } else version(Windows) { 5407 Fiber.yield(); 5408 } 5409 else static assert(0); 5410 } 5411 5412 version(cgi_use_fiber) 5413 void unregisterSource(Socket s) { 5414 version(linux) { 5415 epoll_event evt; 5416 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5417 } else version(Windows) { 5418 // intentionally blank 5419 } 5420 else static assert(0); 5421 } 5422 5423 // it is a class primarily for reference semantics 5424 // I might change this interface 5425 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5426 class BufferedInputRange { 5427 version(Posix) 5428 this(int source, ubyte[] buffer = null) { 5429 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5430 } 5431 5432 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5433 // if they connect but never send stuff to us, we don't want it wasting the process 5434 // so setting a time out 5435 version(cgi_use_fiber) 5436 source.blocking = false; 5437 else 5438 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5439 5440 this.source = source; 5441 if(buffer is null) { 5442 underlyingBuffer = new ubyte[4096]; 5443 this.allowGrowth = true; 5444 } else { 5445 underlyingBuffer = buffer; 5446 this.allowGrowth = allowGrowth; 5447 } 5448 5449 assert(underlyingBuffer.length); 5450 5451 // we assume view.ptr is always inside underlyingBuffer 5452 view = underlyingBuffer[0 .. 0]; 5453 5454 popFront(); // prime 5455 } 5456 5457 version(cgi_use_fiber) { 5458 bool registered; 5459 } 5460 5461 void dispose() { 5462 version(cgi_use_fiber) { 5463 if(registered) 5464 unregisterSource(source); 5465 } 5466 } 5467 5468 /** 5469 A slight difference from regular ranges is you can give it the maximum 5470 number of bytes to consume. 5471 5472 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5473 consume() yourself and use a regular foreach, it will infinitely loop! 5474 5475 The default is to do what a normal range does, and consume the whole buffer 5476 and wait for additional input. 5477 5478 You can also specify 0, to append to the buffer, or any other number 5479 to remove the front n bytes and wait for more. 5480 */ 5481 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5482 if(sourceClosed) 5483 throw new ConnectionClosedException("can't get any more data from a closed source"); 5484 if(!skipConsume) 5485 consume(maxBytesToConsume); 5486 5487 // we might have to grow the buffer 5488 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5489 if(allowGrowth) { 5490 //import std.stdio; writeln("growth"); 5491 auto viewStart = view.ptr - underlyingBuffer.ptr; 5492 size_t growth = 4096; 5493 // make sure we have enough for what we're being asked for 5494 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5495 growth = minBytesToSettleFor - underlyingBuffer.length; 5496 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5497 underlyingBuffer.length += growth; 5498 view = underlyingBuffer[viewStart .. view.length]; 5499 } else 5500 throw new Exception("No room left in the buffer"); 5501 } 5502 5503 do { 5504 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5505 try_again: 5506 auto ret = source.receive(freeSpace); 5507 if(ret == Socket.ERROR) { 5508 if(wouldHaveBlocked()) { 5509 version(cgi_use_fiber) { 5510 registerEventWakeup(®istered, source, WakeupEvent.Read); 5511 goto try_again; 5512 } else { 5513 // gonna treat a timeout here as a close 5514 sourceClosed = true; 5515 return; 5516 } 5517 } 5518 version(Posix) { 5519 import core.stdc.errno; 5520 if(errno == EINTR || errno == EAGAIN) { 5521 goto try_again; 5522 } 5523 if(errno == ECONNRESET) { 5524 sourceClosed = true; 5525 return; 5526 } 5527 } 5528 throw new Exception(lastSocketError); // FIXME 5529 } 5530 if(ret == 0) { 5531 sourceClosed = true; 5532 return; 5533 } 5534 5535 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5536 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5537 //import std.stdio; writeln(cast(string) view); 5538 } while(view.length < minBytesToSettleFor); 5539 } 5540 5541 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5542 /// You might want to idup the data you are consuming if you store it, since it may 5543 /// be overwritten on the new popFront. 5544 /// 5545 /// You do not need to call this if you always want to wait for more data when you 5546 /// consume some. 5547 ubyte[] consume(size_t bytes) { 5548 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5549 view = view[bytes > $ ? $ : bytes .. $]; 5550 if(view.length == 0) { 5551 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5552 /* 5553 writeln("HERE"); 5554 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5555 writeln(cast(string)front); 5556 writeln("DONE"); 5557 */ 5558 } 5559 return front; 5560 } 5561 5562 bool empty() { 5563 return sourceClosed && view.length == 0; 5564 } 5565 5566 ubyte[] front() { 5567 return view; 5568 } 5569 5570 invariant() { 5571 assert(view.ptr >= underlyingBuffer.ptr); 5572 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5573 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5574 } 5575 5576 ubyte[] underlyingBuffer; 5577 bool allowGrowth; 5578 ubyte[] view; 5579 Socket source; 5580 bool sourceClosed; 5581 } 5582 5583 private class FakeSocketForStdin : Socket { 5584 import std.stdio; 5585 5586 this() { 5587 5588 } 5589 5590 private bool closed; 5591 5592 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5593 if(closed) 5594 throw new Exception("Closed"); 5595 return stdin.rawRead(buffer).length; 5596 } 5597 5598 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5599 if(closed) 5600 throw new Exception("Closed"); 5601 stdout.rawWrite(buffer); 5602 return buffer.length; 5603 } 5604 5605 override void close() @trusted scope { 5606 (cast(void delegate() @nogc nothrow) &realClose)(); 5607 } 5608 5609 override void shutdown(SocketShutdown s) { 5610 // FIXME 5611 } 5612 5613 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5614 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5615 5616 override @property @trusted Address remoteAddress() { return null; } 5617 override @property @trusted Address localAddress() { return null; } 5618 5619 void realClose() { 5620 closed = true; 5621 try { 5622 stdin.close(); 5623 stdout.close(); 5624 } catch(Exception e) { 5625 5626 } 5627 } 5628 } 5629 5630 import core.sync.semaphore; 5631 import core.atomic; 5632 5633 /** 5634 To use this thing: 5635 5636 --- 5637 void handler(Socket s) { do something... } 5638 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5639 manager.listen(); 5640 --- 5641 5642 The 4th parameter is optional. 5643 5644 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5645 comes in, you will get control. You can just continue; though to fetch more. 5646 5647 5648 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5649 */ 5650 class ListeningConnectionManager { 5651 Semaphore semaphore; 5652 Socket[256] queue; 5653 shared(ubyte) nextIndexFront; 5654 ubyte nextIndexBack; 5655 shared(int) queueLength; 5656 5657 Socket acceptCancelable() { 5658 version(Posix) { 5659 import core.sys.posix.sys.select; 5660 fd_set read_fds; 5661 FD_ZERO(&read_fds); 5662 int max = 0; 5663 foreach(listener; listeners) { 5664 FD_SET(listener.handle, &read_fds); 5665 if(listener.handle > max) 5666 max = listener.handle; 5667 } 5668 if(cancelfd != -1) { 5669 FD_SET(cancelfd, &read_fds); 5670 if(cancelfd > max) 5671 max = cancelfd; 5672 } 5673 auto ret = select(max + 1, &read_fds, null, null, null); 5674 if(ret == -1) { 5675 import core.stdc.errno; 5676 if(errno == EINTR) 5677 return null; 5678 else 5679 throw new Exception("wtf select"); 5680 } 5681 5682 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5683 return null; 5684 } 5685 5686 foreach(listener; listeners) { 5687 if(FD_ISSET(listener.handle, &read_fds)) 5688 return listener.accept(); 5689 } 5690 5691 return null; 5692 } else { 5693 5694 auto check = new SocketSet(); 5695 5696 keep_looping: 5697 check.reset(); 5698 foreach(listener; listeners) 5699 check.add(listener); 5700 5701 // just to check the stop flag on a kinda busy loop. i hate this FIXME 5702 auto got = Socket.select(check, null, null, 3.seconds); 5703 if(got > 0) 5704 foreach(listener; listeners) 5705 if(check.isSet(listener)) 5706 return listener.accept(); 5707 if(globalStopFlag) 5708 return null; 5709 else 5710 goto keep_looping; 5711 } 5712 } 5713 5714 int defaultNumberOfThreads() { 5715 import std.parallelism; 5716 version(cgi_use_fiber) { 5717 return totalCPUs * 2 + 1; // still chance some will be pointlessly blocked anyway 5718 } else { 5719 // I times 4 here because there's a good chance some will be blocked on i/o. 5720 return totalCPUs * 4; 5721 } 5722 5723 } 5724 5725 void listen() { 5726 shared(int) loopBroken; 5727 5728 version(Posix) { 5729 import core.sys.posix.signal; 5730 signal(SIGPIPE, SIG_IGN); 5731 } 5732 5733 version(linux) { 5734 if(cancelfd == -1) 5735 cancelfd = eventfd(0, 0); 5736 } 5737 5738 version(cgi_no_threads) { 5739 // NEVER USE THIS 5740 // it exists only for debugging and other special occasions 5741 5742 // the thread mode is faster and less likely to stall the whole 5743 // thing when a request is slow 5744 while(!loopBroken && !globalStopFlag) { 5745 auto sn = acceptCancelable(); 5746 if(sn is null) continue; 5747 cloexec(sn); 5748 try { 5749 handler(sn); 5750 } catch(Exception e) { 5751 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5752 sn.close(); 5753 } 5754 } 5755 } else { 5756 5757 if(useFork) { 5758 version(linux) { 5759 //asm { int 3; } 5760 fork(); 5761 } 5762 } 5763 5764 version(cgi_use_fiber) { 5765 5766 version(Windows) { 5767 // please note these are overlapped sockets! so the accept just kicks things off 5768 foreach(listener; listeners) 5769 listener.accept(); 5770 } 5771 5772 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5773 foreach(i, ref thread; threads) { 5774 thread = new WorkerThread(this, handler, cast(int) i); 5775 thread.start(); 5776 } 5777 5778 bool fiber_crash_check() { 5779 bool hasAnyRunning; 5780 foreach(thread; threads) { 5781 if(!thread.isRunning) { 5782 thread.join(); 5783 } else hasAnyRunning = true; 5784 } 5785 5786 return (!hasAnyRunning); 5787 } 5788 5789 5790 while(!globalStopFlag) { 5791 Thread.sleep(1.seconds); 5792 if(fiber_crash_check()) 5793 break; 5794 } 5795 5796 } else { 5797 semaphore = new Semaphore(); 5798 5799 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5800 foreach(i, ref thread; threads) { 5801 thread = new ConnectionThread(this, handler, cast(int) i); 5802 thread.start(); 5803 } 5804 5805 while(!loopBroken && !globalStopFlag) { 5806 Socket sn; 5807 5808 bool crash_check() { 5809 bool hasAnyRunning; 5810 foreach(thread; threads) { 5811 if(!thread.isRunning) { 5812 thread.join(); 5813 } else hasAnyRunning = true; 5814 } 5815 5816 return (!hasAnyRunning); 5817 } 5818 5819 5820 void accept_new_connection() { 5821 sn = acceptCancelable(); 5822 if(sn is null) return; 5823 cloexec(sn); 5824 if(tcp) { 5825 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5826 // on the socket because we do some buffering internally. I think this helps, 5827 // certainly does for small requests, and I think it does for larger ones too 5828 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5829 5830 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5831 } 5832 } 5833 5834 void existing_connection_new_data() { 5835 // wait until a slot opens up 5836 // int waited = 0; 5837 while(queueLength >= queue.length) { 5838 Thread.sleep(1.msecs); 5839 // waited ++; 5840 } 5841 // if(waited) {import std.stdio; writeln(waited);} 5842 synchronized(this) { 5843 queue[nextIndexBack] = sn; 5844 nextIndexBack++; 5845 atomicOp!"+="(queueLength, 1); 5846 } 5847 semaphore.notify(); 5848 } 5849 5850 5851 accept_new_connection(); 5852 if(sn !is null) 5853 existing_connection_new_data(); 5854 else if(sn is null && globalStopFlag) { 5855 foreach(thread; threads) { 5856 semaphore.notify(); 5857 } 5858 Thread.sleep(50.msecs); 5859 } 5860 5861 if(crash_check()) 5862 break; 5863 } 5864 } 5865 5866 // FIXME: i typically stop this with ctrl+c which never 5867 // actually gets here. i need to do a sigint handler. 5868 if(cleanup) 5869 cleanup(); 5870 } 5871 } 5872 5873 //version(linux) 5874 //int epoll_fd; 5875 5876 bool tcp; 5877 void delegate() cleanup; 5878 5879 private void function(Socket) fhandler; 5880 private void dg_handler(Socket s) { 5881 fhandler(s); 5882 } 5883 5884 5885 this(string[] listenSpec, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5886 fhandler = handler; 5887 this(listenSpec, &dg_handler, dropPrivs, useFork, numberOfThreads); 5888 } 5889 this(string[] listenSpec, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5890 string[] host; 5891 ushort[] port; 5892 5893 foreach(spec; listenSpec) { 5894 /+ 5895 The format: 5896 5897 protocol:// 5898 address_spec 5899 5900 Protocol is optional. Must be http, https, scgi, or fastcgi. 5901 5902 address_spec is either: 5903 ipv4 address : port 5904 [ipv6 address] : port 5905 unix:filename 5906 abstract:name 5907 port <which is tcp but on any interface> 5908 +/ 5909 5910 string protocol; 5911 string address_spec; 5912 5913 auto protocolIdx = spec.indexOf("://"); 5914 if(protocolIdx != -1) { 5915 protocol = spec[0 .. protocolIdx]; 5916 address_spec = spec[protocolIdx + "://".length .. $]; 5917 } else { 5918 address_spec = spec; 5919 } 5920 5921 if(address_spec.startsWith("unix:") || address_spec.startsWith("abstract:")) { 5922 host ~= address_spec; 5923 port ~= 0; 5924 } else { 5925 auto idx = address_spec.lastIndexOf(":"); 5926 if(idx == -1) { 5927 host ~= null; 5928 } else { 5929 auto as = address_spec[0 .. idx]; 5930 if(as.length >= 3 && as[0] == '[' && as[$-1] == ']') 5931 as = as[1 .. $-1]; 5932 host ~= as; 5933 } 5934 port ~= address_spec[idx + 1 .. $].to!ushort; 5935 } 5936 5937 } 5938 5939 this(host, port, handler, dropPrivs, useFork, numberOfThreads); 5940 } 5941 5942 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5943 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5944 } 5945 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5946 this([host], [port], handler, dropPrivs, useFork, numberOfThreads); 5947 } 5948 5949 this(string[] host, ushort[] port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5950 fhandler = handler; 5951 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 5952 } 5953 5954 this(string[] host, ushort[] port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5955 assert(host.length == port.length); 5956 5957 this.handler = handler; 5958 this.useFork = useFork; 5959 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 5960 5961 listeners.reserve(host.length); 5962 5963 foreach(i; 0 .. host.length) 5964 if(host[i] == "localhost") { 5965 listeners ~= startListening("127.0.0.1", port[i], tcp, cleanup, 128, dropPrivs); 5966 listeners ~= startListening("::1", port[i], tcp, cleanup, 128, dropPrivs); 5967 } else { 5968 listeners ~= startListening(host[i], port[i], tcp, cleanup, 128, dropPrivs); 5969 } 5970 5971 version(cgi_use_fiber) 5972 if(useFork) { 5973 foreach(listener; listeners) 5974 listener.blocking = false; 5975 } 5976 5977 // this is the UI control thread and thus gets more priority 5978 Thread.getThis.priority = Thread.PRIORITY_MAX; 5979 } 5980 5981 Socket[] listeners; 5982 void delegate(Socket) handler; 5983 5984 immutable bool useFork; 5985 int numberOfThreads; 5986 } 5987 5988 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 5989 Socket listener; 5990 if(host.startsWith("unix:")) { 5991 version(Posix) { 5992 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5993 cloexec(listener); 5994 string filename = host["unix:".length .. $].idup; 5995 listener.bind(new UnixAddress(filename)); 5996 cleanup = delegate() { 5997 listener.close(); 5998 import std.file; 5999 remove(filename); 6000 }; 6001 tcp = false; 6002 } else { 6003 throw new Exception("unix sockets not supported on this system"); 6004 } 6005 } else if(host.startsWith("abstract:")) { 6006 version(linux) { 6007 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 6008 cloexec(listener); 6009 string filename = "\0" ~ host["abstract:".length .. $]; 6010 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 6011 listener.bind(new UnixAddress(filename)); 6012 tcp = false; 6013 } else { 6014 throw new Exception("abstract unix sockets not supported on this system"); 6015 } 6016 } else { 6017 auto address = host.length ? parseAddress(host, port) : new InternetAddress(port); 6018 version(cgi_use_fiber) { 6019 version(Windows) 6020 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 6021 else 6022 listener = new Socket(address.addressFamily, SocketType.STREAM); 6023 } else { 6024 listener = new Socket(address.addressFamily, SocketType.STREAM); 6025 } 6026 cloexec(listener); 6027 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 6028 if(address.addressFamily == AddressFamily.INET6) 6029 listener.setOption(SocketOptionLevel.IPV6, SocketOption.IPV6_V6ONLY, true); 6030 listener.bind(address); 6031 cleanup = delegate() { 6032 listener.close(); 6033 }; 6034 tcp = true; 6035 } 6036 6037 listener.listen(backQueue); 6038 6039 if (dropPrivs !is null) // can be null, backwards compatibility 6040 dropPrivs(); 6041 6042 return listener; 6043 } 6044 6045 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 6046 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 6047 if(data.length == 0) return; 6048 ptrdiff_t amount; 6049 //import std.stdio; writeln("***",cast(string) data,"///"); 6050 do { 6051 amount = s.send(data); 6052 if(amount == Socket.ERROR) { 6053 version(cgi_use_fiber) { 6054 if(wouldHaveBlocked()) { 6055 bool registered = true; 6056 registerEventWakeup(®istered, s, WakeupEvent.Write); 6057 continue; 6058 } 6059 } 6060 throw new ConnectionException(s, lastSocketError, file, line); 6061 } 6062 assert(amount > 0); 6063 6064 data = data[amount .. $]; 6065 } while(data.length); 6066 } 6067 6068 class ConnectionException : Exception { 6069 Socket socket; 6070 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 6071 this.socket = s; 6072 super(msg, file, line); 6073 } 6074 } 6075 6076 class HttpVersionNotSupportedException : Exception { 6077 this(string file = __FILE__, size_t line = __LINE__) { 6078 super("HTTP Version Not Supported", file, line); 6079 } 6080 } 6081 6082 alias void delegate(Socket) CMT; 6083 6084 import core.thread; 6085 /+ 6086 cgi.d now uses a hybrid of event i/o and threads at the top level. 6087 6088 Top level thread is responsible for accepting sockets and selecting on them. 6089 6090 It then indicates to a child that a request is pending, and any random worker 6091 thread that is free handles it. It goes into blocking mode and handles that 6092 http request to completion. 6093 6094 At that point, it goes back into the waiting queue. 6095 6096 6097 This concept is only implemented on Linux. On all other systems, it still 6098 uses the worker threads and semaphores (which is perfectly fine for a lot of 6099 things! Just having a great number of keep-alive connections will break that.) 6100 6101 6102 So the algorithm is: 6103 6104 select(accept, event, pending) 6105 if accept -> send socket to free thread, if any. if not, add socket to queue 6106 if event -> send the signaling thread a socket from the queue, if not, mark it free 6107 - event might block until it can be *written* to. it is a fifo sending socket fds! 6108 6109 A worker only does one http request at a time, then signals its availability back to the boss. 6110 6111 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 6112 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 6113 actual FD will not be kept out here. 6114 6115 So: 6116 queue = sockets we know are ready to read now, but no worker thread is available 6117 idle list = worker threads not doing anything else. they signal back and forth 6118 6119 the workers all read off the event fd. This is the semaphore wait 6120 6121 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 6122 it puts it in the queue and writes to the event fd. 6123 6124 The child could put the socket back in the epoll thing itself. 6125 6126 The child needs to be able to gracefully handle being given a socket that just closed with no work. 6127 +/ 6128 class ConnectionThread : Thread { 6129 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6130 this.lcm = lcm; 6131 this.dg = dg; 6132 this.myThreadNumber = myThreadNumber; 6133 super(&run); 6134 } 6135 6136 void run() { 6137 while(true) { 6138 // so if there's a bunch of idle keep-alive connections, it can 6139 // consume all the worker threads... just sitting there. 6140 lcm.semaphore.wait(); 6141 if(globalStopFlag) 6142 return; 6143 Socket socket; 6144 synchronized(lcm) { 6145 auto idx = lcm.nextIndexFront; 6146 socket = lcm.queue[idx]; 6147 lcm.queue[idx] = null; 6148 atomicOp!"+="(lcm.nextIndexFront, 1); 6149 atomicOp!"-="(lcm.queueLength, 1); 6150 } 6151 try { 6152 //import std.stdio; writeln(myThreadNumber, " taking it"); 6153 dg(socket); 6154 /+ 6155 if(socket.isAlive) { 6156 // process it more later 6157 version(linux) { 6158 import core.sys.linux.epoll; 6159 epoll_event ev; 6160 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6161 ev.data.fd = socket.handle; 6162 import std.stdio; writeln("adding"); 6163 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 6164 if(errno == EEXIST) { 6165 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 6166 ev.data.fd = socket.handle; 6167 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 6168 throw new Exception("epoll_ctl " ~ to!string(errno)); 6169 } else 6170 throw new Exception("epoll_ctl " ~ to!string(errno)); 6171 } 6172 //import std.stdio; writeln("keep alive"); 6173 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 6174 __traits(getMember, socket, "sock") = cast(socket_t) -1; 6175 } else { 6176 continue; // hope it times out in a reasonable amount of time... 6177 } 6178 } 6179 +/ 6180 } catch(ConnectionClosedException e) { 6181 // can just ignore this, it is fairly normal 6182 socket.close(); 6183 } catch(Throwable e) { 6184 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 6185 socket.close(); 6186 } 6187 } 6188 } 6189 6190 ListeningConnectionManager lcm; 6191 CMT dg; 6192 int myThreadNumber; 6193 } 6194 6195 version(cgi_use_fiber) 6196 class WorkerThread : Thread { 6197 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 6198 this.lcm = lcm; 6199 this.dg = dg; 6200 this.myThreadNumber = myThreadNumber; 6201 super(&run); 6202 } 6203 6204 version(Windows) 6205 void run() { 6206 auto timeout = INFINITE; 6207 PseudoblockingOverlappedSocket key; 6208 OVERLAPPED* overlapped; 6209 DWORD bytes; 6210 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 6211 if(key is null) 6212 continue; 6213 key.lastAnswer = bytes; 6214 if(key.fiber) { 6215 key.fiber.proceed(); 6216 } else { 6217 // we have a new connection, issue the first receive on it and issue the next accept 6218 6219 auto sn = key.accepted; 6220 6221 key.accept(); 6222 6223 cloexec(sn); 6224 if(lcm.tcp) { 6225 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6226 // on the socket because we do some buffering internally. I think this helps, 6227 // certainly does for small requests, and I think it does for larger ones too 6228 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6229 6230 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6231 } 6232 6233 dg(sn); 6234 } 6235 } 6236 //SleepEx(INFINITE, TRUE); 6237 } 6238 6239 version(linux) 6240 void run() { 6241 6242 import core.sys.linux.epoll; 6243 epfd = epoll_create1(EPOLL_CLOEXEC); 6244 if(epfd == -1) 6245 throw new Exception("epoll_create1 " ~ to!string(errno)); 6246 scope(exit) { 6247 import core.sys.posix.unistd; 6248 close(epfd); 6249 } 6250 6251 { 6252 epoll_event ev; 6253 ev.events = EPOLLIN; 6254 ev.data.fd = cancelfd; 6255 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 6256 } 6257 6258 foreach(listener; lcm.listeners) { 6259 epoll_event ev; 6260 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 6261 ev.data.fd = listener.handle; 6262 if(epoll_ctl(epfd, EPOLL_CTL_ADD, listener.handle, &ev) == -1) 6263 throw new Exception("epoll_ctl " ~ to!string(errno)); 6264 } 6265 6266 6267 6268 while(!globalStopFlag) { 6269 Socket sn; 6270 6271 epoll_event[64] events; 6272 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 6273 if(nfds == -1) { 6274 if(errno == EINTR) 6275 continue; 6276 throw new Exception("epoll_wait " ~ to!string(errno)); 6277 } 6278 6279 outer: foreach(idx; 0 .. nfds) { 6280 auto flags = events[idx].events; 6281 6282 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 6283 globalStopFlag = true; 6284 //import std.stdio; writeln("exit heard"); 6285 break; 6286 } else { 6287 foreach(listener; lcm.listeners) { 6288 if(cast(size_t) events[idx].data.ptr == cast(size_t) listener.handle) { 6289 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 6290 // this try/catch is because it is set to non-blocking mode 6291 // and Phobos' stupid api throws an exception instead of returning 6292 // if it would block. Why would it block? because a forked process 6293 // might have beat us to it, but the wakeup event thundered our herds. 6294 try 6295 sn = listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 6296 catch(SocketAcceptException e) { continue outer; } 6297 6298 cloexec(sn); 6299 if(lcm.tcp) { 6300 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 6301 // on the socket because we do some buffering internally. I think this helps, 6302 // certainly does for small requests, and I think it does for larger ones too 6303 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 6304 6305 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 6306 } 6307 6308 dg(sn); 6309 continue outer; 6310 } else { 6311 // writeln(events[idx].data.ptr); 6312 } 6313 } 6314 6315 if(cast(size_t) events[idx].data.ptr < 1024) { 6316 throw arsd.core.ArsdException!"this doesn't look like a fiber pointer... "(cast(size_t) events[idx].data.ptr); 6317 } 6318 auto fiber = cast(CgiFiber) events[idx].data.ptr; 6319 fiber.proceed(); 6320 } 6321 } 6322 } 6323 } 6324 6325 ListeningConnectionManager lcm; 6326 CMT dg; 6327 int myThreadNumber; 6328 } 6329 6330 6331 /* Done with network helper */ 6332 6333 /* Helpers for doing temporary files. Used both here and in web.d */ 6334 6335 version(Windows) { 6336 import core.sys.windows.windows; 6337 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 6338 alias GetTempPathW GetTempPath; 6339 } 6340 6341 version(Posix) { 6342 static import linux = core.sys.posix.unistd; 6343 } 6344 6345 string getTempDirectory() { 6346 string path; 6347 version(Windows) { 6348 wchar[1024] buffer; 6349 auto len = GetTempPath(1024, buffer.ptr); 6350 if(len == 0) 6351 throw new Exception("couldn't find a temporary path"); 6352 6353 auto b = buffer[0 .. len]; 6354 6355 path = to!string(b); 6356 } else 6357 path = "/tmp/"; 6358 6359 return path; 6360 } 6361 6362 6363 // I like std.date. These functions help keep my old code and data working with phobos changing. 6364 6365 long sysTimeToDTime(in SysTime sysTime) { 6366 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 6367 } 6368 6369 long dateTimeToDTime(in DateTime dt) { 6370 return sysTimeToDTime(cast(SysTime) dt); 6371 } 6372 6373 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 6374 return sysTimeToDTime(Clock.currTime(UTC())); 6375 } 6376 6377 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 6378 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 6379 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 6380 return SysTime(hnsecs, tz); 6381 } 6382 6383 6384 6385 // this is a helper to read HTTP transfer-encoding: chunked responses 6386 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 6387 immutable(ubyte)[] ret; 6388 6389 another_chunk: 6390 // If here, we are at the beginning of a chunk. 6391 auto a = ir.front(); 6392 int chunkSize; 6393 int loc = locationOf(a, "\r\n"); 6394 while(loc == -1) { 6395 ir.popFront(); 6396 a = ir.front(); 6397 loc = locationOf(a, "\r\n"); 6398 } 6399 6400 string hex; 6401 hex = ""; 6402 for(int i = 0; i < loc; i++) { 6403 char c = a[i]; 6404 if(c >= 'A' && c <= 'Z') 6405 c += 0x20; 6406 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 6407 hex ~= c; 6408 } else { 6409 break; 6410 } 6411 } 6412 6413 assert(hex.length); 6414 6415 int power = 1; 6416 int size = 0; 6417 foreach(cc1; retro(hex)) { 6418 dchar cc = cc1; 6419 if(cc >= 'a' && cc <= 'z') 6420 cc -= 0x20; 6421 int val = 0; 6422 if(cc >= '0' && cc <= '9') 6423 val = cc - '0'; 6424 else 6425 val = cc - 'A' + 10; 6426 6427 size += power * val; 6428 power *= 16; 6429 } 6430 6431 chunkSize = size; 6432 assert(size >= 0); 6433 6434 if(loc + 2 > a.length) { 6435 ir.popFront(0, a.length + loc + 2); 6436 a = ir.front(); 6437 } 6438 6439 a = ir.consume(loc + 2); 6440 6441 if(chunkSize == 0) { // we're done with the response 6442 // if we got here, will change must be true.... 6443 more_footers: 6444 loc = locationOf(a, "\r\n"); 6445 if(loc == -1) { 6446 ir.popFront(); 6447 a = ir.front; 6448 goto more_footers; 6449 } else { 6450 assert(loc == 0); 6451 ir.consume(loc + 2); 6452 goto finish; 6453 } 6454 } else { 6455 // if we got here, will change must be true.... 6456 if(a.length < chunkSize + 2) { 6457 ir.popFront(0, chunkSize + 2); 6458 a = ir.front(); 6459 } 6460 6461 ret ~= (a[0..chunkSize]); 6462 6463 if(!(a.length > chunkSize + 2)) { 6464 ir.popFront(0, chunkSize + 2); 6465 a = ir.front(); 6466 } 6467 assert(a[chunkSize] == 13); 6468 assert(a[chunkSize+1] == 10); 6469 a = ir.consume(chunkSize + 2); 6470 chunkSize = 0; 6471 goto another_chunk; 6472 } 6473 6474 finish: 6475 return ret; 6476 } 6477 6478 // I want to be able to get data from multiple sources the same way... 6479 interface ByChunkRange { 6480 bool empty(); 6481 void popFront(); 6482 const(ubyte)[] front(); 6483 } 6484 6485 ByChunkRange byChunk(const(ubyte)[] data) { 6486 return new class ByChunkRange { 6487 override bool empty() { 6488 return !data.length; 6489 } 6490 6491 override void popFront() { 6492 if(data.length > 4096) 6493 data = data[4096 .. $]; 6494 else 6495 data = null; 6496 } 6497 6498 override const(ubyte)[] front() { 6499 return data[0 .. $ > 4096 ? 4096 : $]; 6500 } 6501 }; 6502 } 6503 6504 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 6505 const(ubyte)[] f; 6506 6507 f = ir.front; 6508 if(f.length > atMost) 6509 f = f[0 .. atMost]; 6510 6511 return new class ByChunkRange { 6512 override bool empty() { 6513 return atMost == 0; 6514 } 6515 6516 override const(ubyte)[] front() { 6517 return f; 6518 } 6519 6520 override void popFront() { 6521 ir.consume(f.length); 6522 atMost -= f.length; 6523 auto a = ir.front(); 6524 6525 if(a.length <= atMost) { 6526 f = a; 6527 atMost -= a.length; 6528 a = ir.consume(a.length); 6529 if(atMost != 0) 6530 ir.popFront(); 6531 if(f.length == 0) { 6532 f = ir.front(); 6533 } 6534 } else { 6535 // we actually have *more* here than we need.... 6536 f = a[0..atMost]; 6537 atMost = 0; 6538 ir.consume(atMost); 6539 } 6540 } 6541 }; 6542 } 6543 6544 version(cgi_with_websocket) { 6545 // http://tools.ietf.org/html/rfc6455 6546 6547 /++ 6548 WEBSOCKET SUPPORT: 6549 6550 Full example: 6551 --- 6552 import arsd.cgi; 6553 6554 void websocketEcho(Cgi cgi) { 6555 if(cgi.websocketRequested()) { 6556 if(cgi.origin != "http://arsdnet.net") 6557 throw new Exception("bad origin"); 6558 auto websocket = cgi.acceptWebsocket(); 6559 6560 websocket.send("hello"); 6561 websocket.send(" world!"); 6562 6563 auto msg = websocket.recv(); 6564 while(msg.opcode != WebSocketOpcode.close) { 6565 if(msg.opcode == WebSocketOpcode.text) { 6566 websocket.send(msg.textData); 6567 } else if(msg.opcode == WebSocketOpcode.binary) { 6568 websocket.send(msg.data); 6569 } 6570 6571 msg = websocket.recv(); 6572 } 6573 6574 websocket.close(); 6575 } else { 6576 cgi.write("You are loading the websocket endpoint in a browser instead of a websocket client. Use a websocket client on this url instead.\n", true); 6577 } 6578 } 6579 6580 mixin GenericMain!websocketEcho; 6581 --- 6582 +/ 6583 6584 class WebSocket { 6585 Cgi cgi; 6586 6587 private this(Cgi cgi) { 6588 this.cgi = cgi; 6589 6590 Socket socket = cgi.idlol.source; 6591 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6592 } 6593 6594 // returns true if data available, false if it timed out 6595 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6596 if(!waitForNextMessageWouldBlock()) 6597 return true; 6598 if(isDataPending(timeout)) 6599 return true; // this is kinda a lie. 6600 6601 return false; 6602 } 6603 6604 public bool lowLevelReceive() { 6605 auto bfr = cgi.idlol; 6606 top: 6607 auto got = bfr.front; 6608 if(got.length) { 6609 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6610 receiveBuffer.length += receiveBufferUsedLength + got.length; 6611 6612 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6613 receiveBufferUsedLength += got.length; 6614 bfr.consume(got.length); 6615 6616 return true; 6617 } 6618 6619 if(bfr.sourceClosed) 6620 return false; 6621 6622 bfr.popFront(0); 6623 if(bfr.sourceClosed) 6624 return false; 6625 goto top; 6626 } 6627 6628 6629 bool isDataPending(Duration timeout = 0.seconds) { 6630 Socket socket = cgi.idlol.source; 6631 6632 auto check = new SocketSet(); 6633 check.add(socket); 6634 6635 auto got = Socket.select(check, null, null, timeout); 6636 if(got > 0) 6637 return true; 6638 return false; 6639 } 6640 6641 // note: this blocks 6642 WebSocketFrame recv() { 6643 return waitForNextMessage(); 6644 } 6645 6646 6647 6648 6649 private void llclose() { 6650 cgi.close(); 6651 } 6652 6653 private void llsend(ubyte[] data) { 6654 cgi.write(data); 6655 cgi.flush(); 6656 } 6657 6658 void unregisterActiveSocket(WebSocket) {} 6659 6660 /* copy/paste section { */ 6661 6662 private int readyState_; 6663 private ubyte[] receiveBuffer; 6664 private size_t receiveBufferUsedLength; 6665 6666 private Config config; 6667 6668 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6669 enum OPEN = 1; /// The connection is open and ready to communicate. 6670 enum CLOSING = 2; /// The connection is in the process of closing. 6671 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6672 6673 /++ 6674 6675 +/ 6676 /// Group: foundational 6677 static struct Config { 6678 /++ 6679 These control the size of the receive buffer. 6680 6681 It starts at the initial size, will temporarily 6682 balloon up to the maximum size, and will reuse 6683 a buffer up to the likely size. 6684 6685 Anything larger than the maximum size will cause 6686 the connection to be aborted and an exception thrown. 6687 This is to protect you against a peer trying to 6688 exhaust your memory, while keeping the user-level 6689 processing simple. 6690 +/ 6691 size_t initialReceiveBufferSize = 4096; 6692 size_t likelyReceiveBufferSize = 4096; /// ditto 6693 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6694 6695 /++ 6696 Maximum combined size of a message. 6697 +/ 6698 size_t maximumMessageSize = 10 * 1024 * 1024; 6699 6700 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6701 string origin; /// Origin URL to send with the handshake, if desired. 6702 string protocol; /// the protocol header, if desired. 6703 6704 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6705 } 6706 6707 /++ 6708 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6709 +/ 6710 int readyState() { 6711 return readyState_; 6712 } 6713 6714 /++ 6715 Closes the connection, sending a graceful teardown message to the other side. 6716 +/ 6717 /// Group: foundational 6718 void close(int code = 0, string reason = null) 6719 //in (reason.length < 123) 6720 in { assert(reason.length < 123); } do 6721 { 6722 if(readyState_ != OPEN) 6723 return; // it cool, we done 6724 WebSocketFrame wss; 6725 wss.fin = true; 6726 wss.opcode = WebSocketOpcode.close; 6727 wss.data = cast(ubyte[]) reason.dup; 6728 wss.send(&llsend); 6729 6730 readyState_ = CLOSING; 6731 6732 llclose(); 6733 } 6734 6735 /++ 6736 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6737 +/ 6738 /// Group: foundational 6739 void ping() { 6740 WebSocketFrame wss; 6741 wss.fin = true; 6742 wss.opcode = WebSocketOpcode.ping; 6743 wss.send(&llsend); 6744 } 6745 6746 // automatically handled.... 6747 void pong() { 6748 WebSocketFrame wss; 6749 wss.fin = true; 6750 wss.opcode = WebSocketOpcode.pong; 6751 wss.send(&llsend); 6752 } 6753 6754 /++ 6755 Sends a text message through the websocket. 6756 +/ 6757 /// Group: foundational 6758 void send(in char[] textData) { 6759 WebSocketFrame wss; 6760 wss.fin = true; 6761 wss.opcode = WebSocketOpcode.text; 6762 wss.data = cast(ubyte[]) textData.dup; 6763 wss.send(&llsend); 6764 } 6765 6766 /++ 6767 Sends a binary message through the websocket. 6768 +/ 6769 /// Group: foundational 6770 void send(in ubyte[] binaryData) { 6771 WebSocketFrame wss; 6772 wss.fin = true; 6773 wss.opcode = WebSocketOpcode.binary; 6774 wss.data = cast(ubyte[]) binaryData.dup; 6775 wss.send(&llsend); 6776 } 6777 6778 /++ 6779 Waits for and returns the next complete message on the socket. 6780 6781 Note that the onmessage function is still called, right before 6782 this returns. 6783 +/ 6784 /// Group: blocking_api 6785 public WebSocketFrame waitForNextMessage() { 6786 do { 6787 auto m = processOnce(); 6788 if(m.populated) 6789 return m; 6790 } while(lowLevelReceive()); 6791 6792 throw new ConnectionClosedException("Websocket receive timed out"); 6793 //return WebSocketFrame.init; // FIXME? maybe. 6794 } 6795 6796 /++ 6797 Tells if [waitForNextMessage] would block. 6798 +/ 6799 /// Group: blocking_api 6800 public bool waitForNextMessageWouldBlock() { 6801 checkAgain: 6802 if(isMessageBuffered()) 6803 return false; 6804 if(!isDataPending()) 6805 return true; 6806 while(isDataPending()) 6807 lowLevelReceive(); 6808 goto checkAgain; 6809 } 6810 6811 /++ 6812 Is there a message in the buffer already? 6813 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6814 If `false`, check [isDataPending] as the next step. 6815 +/ 6816 /// Group: blocking_api 6817 public bool isMessageBuffered() { 6818 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6819 auto s = d; 6820 if(d.length) { 6821 auto orig = d; 6822 auto m = WebSocketFrame.read(d); 6823 // that's how it indicates that it needs more data 6824 if(d !is orig) 6825 return true; 6826 } 6827 6828 return false; 6829 } 6830 6831 private ubyte continuingType; 6832 private ubyte[] continuingData; 6833 //private size_t continuingDataLength; 6834 6835 private WebSocketFrame processOnce() { 6836 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6837 auto s = d; 6838 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6839 WebSocketFrame m; 6840 if(d.length) { 6841 auto orig = d; 6842 m = WebSocketFrame.read(d); 6843 // that's how it indicates that it needs more data 6844 if(d is orig) 6845 return WebSocketFrame.init; 6846 m.unmaskInPlace(); 6847 switch(m.opcode) { 6848 case WebSocketOpcode.continuation: 6849 if(continuingData.length + m.data.length > config.maximumMessageSize) 6850 throw new Exception("message size exceeded"); 6851 6852 continuingData ~= m.data; 6853 if(m.fin) { 6854 if(ontextmessage) 6855 ontextmessage(cast(char[]) continuingData); 6856 if(onbinarymessage) 6857 onbinarymessage(continuingData); 6858 6859 continuingData = null; 6860 } 6861 break; 6862 case WebSocketOpcode.text: 6863 if(m.fin) { 6864 if(ontextmessage) 6865 ontextmessage(m.textData); 6866 } else { 6867 continuingType = m.opcode; 6868 //continuingDataLength = 0; 6869 continuingData = null; 6870 continuingData ~= m.data; 6871 } 6872 break; 6873 case WebSocketOpcode.binary: 6874 if(m.fin) { 6875 if(onbinarymessage) 6876 onbinarymessage(m.data); 6877 } else { 6878 continuingType = m.opcode; 6879 //continuingDataLength = 0; 6880 continuingData = null; 6881 continuingData ~= m.data; 6882 } 6883 break; 6884 case WebSocketOpcode.close: 6885 readyState_ = CLOSED; 6886 if(onclose) 6887 onclose(); 6888 6889 unregisterActiveSocket(this); 6890 break; 6891 case WebSocketOpcode.ping: 6892 pong(); 6893 break; 6894 case WebSocketOpcode.pong: 6895 // just really references it is still alive, nbd. 6896 break; 6897 default: // ignore though i could and perhaps should throw too 6898 } 6899 } 6900 6901 // the recv thing can be invalidated so gotta copy it over ugh 6902 if(d.length) { 6903 m.data = m.data.dup(); 6904 } 6905 6906 import core.stdc.string; 6907 memmove(receiveBuffer.ptr, d.ptr, d.length); 6908 receiveBufferUsedLength = d.length; 6909 6910 return m; 6911 } 6912 6913 private void autoprocess() { 6914 // FIXME 6915 do { 6916 processOnce(); 6917 } while(lowLevelReceive()); 6918 } 6919 6920 6921 void delegate() onclose; /// 6922 void delegate() onerror; /// 6923 void delegate(in char[]) ontextmessage; /// 6924 void delegate(in ubyte[]) onbinarymessage; /// 6925 void delegate() onopen; /// 6926 6927 /++ 6928 6929 +/ 6930 /// Group: browser_api 6931 void onmessage(void delegate(in char[]) dg) { 6932 ontextmessage = dg; 6933 } 6934 6935 /// ditto 6936 void onmessage(void delegate(in ubyte[]) dg) { 6937 onbinarymessage = dg; 6938 } 6939 6940 /* } end copy/paste */ 6941 6942 6943 } 6944 6945 /++ 6946 Returns true if the request headers are asking for a websocket upgrade. 6947 6948 If this returns true, and you want to accept it, call [acceptWebsocket]. 6949 +/ 6950 bool websocketRequested(Cgi cgi) { 6951 return 6952 "sec-websocket-key" in cgi.requestHeaders 6953 && 6954 "connection" in cgi.requestHeaders && 6955 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 6956 && 6957 "upgrade" in cgi.requestHeaders && 6958 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 6959 ; 6960 } 6961 6962 /++ 6963 If [websocketRequested], you can call this to accept it and upgrade the connection. It returns the new [WebSocket] object you use for future communication on this connection; the `cgi` object should no longer be used. 6964 +/ 6965 WebSocket acceptWebsocket(Cgi cgi) { 6966 assert(!cgi.closed); 6967 assert(!cgi.outputtedResponseData); 6968 cgi.setResponseStatus("101 Switching Protocols"); 6969 cgi.header("Upgrade: WebSocket"); 6970 cgi.header("Connection: upgrade"); 6971 6972 string key = cgi.requestHeaders["sec-websocket-key"]; 6973 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 6974 6975 import std.digest.sha; 6976 auto hash = sha1Of(key); 6977 auto accept = Base64.encode(hash); 6978 6979 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 6980 6981 cgi.websocketMode = true; 6982 cgi.write(""); 6983 6984 cgi.flush(); 6985 6986 return new WebSocket(cgi); 6987 } 6988 6989 // FIXME get websocket to work on other modes, not just embedded_httpd 6990 6991 /* copy/paste in http2.d { */ 6992 enum WebSocketOpcode : ubyte { 6993 continuation = 0, 6994 text = 1, 6995 binary = 2, 6996 // 3, 4, 5, 6, 7 RESERVED 6997 close = 8, 6998 ping = 9, 6999 pong = 10, 7000 // 11,12,13,14,15 RESERVED 7001 } 7002 7003 public struct WebSocketFrame { 7004 private bool populated; 7005 bool fin; 7006 bool rsv1; 7007 bool rsv2; 7008 bool rsv3; 7009 WebSocketOpcode opcode; // 4 bits 7010 bool masked; 7011 ubyte lengthIndicator; // don't set this when building one to send 7012 ulong realLength; // don't use when sending 7013 ubyte[4] maskingKey; // don't set this when sending 7014 ubyte[] data; 7015 7016 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 7017 WebSocketFrame msg; 7018 msg.fin = true; 7019 msg.opcode = opcode; 7020 msg.data = cast(ubyte[]) data.dup; 7021 7022 return msg; 7023 } 7024 7025 private void send(scope void delegate(ubyte[]) llsend) { 7026 ubyte[64] headerScratch; 7027 int headerScratchPos = 0; 7028 7029 realLength = data.length; 7030 7031 { 7032 ubyte b1; 7033 b1 |= cast(ubyte) opcode; 7034 b1 |= rsv3 ? (1 << 4) : 0; 7035 b1 |= rsv2 ? (1 << 5) : 0; 7036 b1 |= rsv1 ? (1 << 6) : 0; 7037 b1 |= fin ? (1 << 7) : 0; 7038 7039 headerScratch[0] = b1; 7040 headerScratchPos++; 7041 } 7042 7043 { 7044 headerScratchPos++; // we'll set header[1] at the end of this 7045 auto rlc = realLength; 7046 ubyte b2; 7047 b2 |= masked ? (1 << 7) : 0; 7048 7049 assert(headerScratchPos == 2); 7050 7051 if(realLength > 65535) { 7052 // use 64 bit length 7053 b2 |= 0x7f; 7054 7055 // FIXME: double check endinaness 7056 foreach(i; 0 .. 8) { 7057 headerScratch[2 + 7 - i] = rlc & 0x0ff; 7058 rlc >>>= 8; 7059 } 7060 7061 headerScratchPos += 8; 7062 } else if(realLength > 125) { 7063 // use 16 bit length 7064 b2 |= 0x7e; 7065 7066 // FIXME: double check endinaness 7067 foreach(i; 0 .. 2) { 7068 headerScratch[2 + 1 - i] = rlc & 0x0ff; 7069 rlc >>>= 8; 7070 } 7071 7072 headerScratchPos += 2; 7073 } else { 7074 // use 7 bit length 7075 b2 |= realLength & 0b_0111_1111; 7076 } 7077 7078 headerScratch[1] = b2; 7079 } 7080 7081 //assert(!masked, "masking key not properly implemented"); 7082 if(masked) { 7083 // FIXME: randomize this 7084 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 7085 headerScratchPos += 4; 7086 7087 // we'll just mask it in place... 7088 int keyIdx = 0; 7089 foreach(i; 0 .. data.length) { 7090 data[i] = data[i] ^ maskingKey[keyIdx]; 7091 if(keyIdx == 3) 7092 keyIdx = 0; 7093 else 7094 keyIdx++; 7095 } 7096 } 7097 7098 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 7099 llsend(headerScratch[0 .. headerScratchPos]); 7100 llsend(data); 7101 } 7102 7103 static WebSocketFrame read(ref ubyte[] d) { 7104 WebSocketFrame msg; 7105 7106 auto orig = d; 7107 7108 WebSocketFrame needsMoreData() { 7109 d = orig; 7110 return WebSocketFrame.init; 7111 } 7112 7113 if(d.length < 2) 7114 return needsMoreData(); 7115 7116 ubyte b = d[0]; 7117 7118 msg.populated = true; 7119 7120 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 7121 b >>= 4; 7122 msg.rsv3 = b & 0x01; 7123 b >>= 1; 7124 msg.rsv2 = b & 0x01; 7125 b >>= 1; 7126 msg.rsv1 = b & 0x01; 7127 b >>= 1; 7128 msg.fin = b & 0x01; 7129 7130 b = d[1]; 7131 msg.masked = (b & 0b1000_0000) ? true : false; 7132 msg.lengthIndicator = b & 0b0111_1111; 7133 7134 d = d[2 .. $]; 7135 7136 if(msg.lengthIndicator == 0x7e) { 7137 // 16 bit length 7138 msg.realLength = 0; 7139 7140 if(d.length < 2) return needsMoreData(); 7141 7142 foreach(i; 0 .. 2) { 7143 msg.realLength |= d[0] << ((1-i) * 8); 7144 d = d[1 .. $]; 7145 } 7146 } else if(msg.lengthIndicator == 0x7f) { 7147 // 64 bit length 7148 msg.realLength = 0; 7149 7150 if(d.length < 8) return needsMoreData(); 7151 7152 foreach(i; 0 .. 8) { 7153 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 7154 d = d[1 .. $]; 7155 } 7156 } else { 7157 // 7 bit length 7158 msg.realLength = msg.lengthIndicator; 7159 } 7160 7161 if(msg.masked) { 7162 7163 if(d.length < 4) return needsMoreData(); 7164 7165 msg.maskingKey = d[0 .. 4]; 7166 d = d[4 .. $]; 7167 } 7168 7169 if(msg.realLength > d.length) { 7170 return needsMoreData(); 7171 } 7172 7173 msg.data = d[0 .. cast(size_t) msg.realLength]; 7174 d = d[cast(size_t) msg.realLength .. $]; 7175 7176 return msg; 7177 } 7178 7179 void unmaskInPlace() { 7180 if(this.masked) { 7181 int keyIdx = 0; 7182 foreach(i; 0 .. this.data.length) { 7183 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 7184 if(keyIdx == 3) 7185 keyIdx = 0; 7186 else 7187 keyIdx++; 7188 } 7189 } 7190 } 7191 7192 char[] textData() { 7193 return cast(char[]) data; 7194 } 7195 } 7196 /* } */ 7197 } 7198 7199 7200 version(Windows) 7201 { 7202 version(CRuntime_DigitalMars) 7203 { 7204 extern(C) int setmode(int, int) nothrow @nogc; 7205 } 7206 else version(CRuntime_Microsoft) 7207 { 7208 extern(C) int _setmode(int, int) nothrow @nogc; 7209 alias setmode = _setmode; 7210 } 7211 else static assert(0); 7212 } 7213 7214 version(Posix) { 7215 import core.sys.posix.unistd; 7216 version(CRuntime_Musl) {} else { 7217 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 7218 } 7219 } 7220 7221 7222 // FIXME: these aren't quite public yet. 7223 //private: 7224 7225 // template for laziness 7226 void startAddonServer()(string arg) { 7227 version(OSX) { 7228 assert(0, "Not implemented"); 7229 } else version(linux) { 7230 import core.sys.posix.unistd; 7231 pid_t pid; 7232 const(char)*[16] args; 7233 args[0] = "ARSD_CGI_ADDON_SERVER"; 7234 args[1] = arg.ptr; 7235 posix_spawn(&pid, "/proc/self/exe", 7236 null, 7237 null, 7238 args.ptr, 7239 null // env 7240 ); 7241 } else version(Windows) { 7242 wchar[2048] filename; 7243 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 7244 if(len == 0 || len == filename.length) 7245 throw new Exception("could not get process name to start helper server"); 7246 7247 STARTUPINFOW startupInfo; 7248 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 7249 PROCESS_INFORMATION processInfo; 7250 7251 import std.utf; 7252 7253 // I *MIGHT* need to run it as a new job or a service... 7254 auto ret = CreateProcessW( 7255 filename.ptr, 7256 toUTF16z(arg), 7257 null, // process attributes 7258 null, // thread attributes 7259 false, // inherit handles 7260 0, // creation flags 7261 null, // environment 7262 null, // working directory 7263 &startupInfo, 7264 &processInfo 7265 ); 7266 7267 if(!ret) 7268 throw new Exception("create process failed"); 7269 7270 // when done with those, if we set them 7271 /* 7272 CloseHandle(hStdInput); 7273 CloseHandle(hStdOutput); 7274 CloseHandle(hStdError); 7275 */ 7276 7277 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 7278 } 7279 7280 // template for laziness 7281 /* 7282 The websocket server is a single-process, single-thread, event 7283 I/O thing. It is passed websockets from other CGI processes 7284 and is then responsible for handling their messages and responses. 7285 Note that the CGI process is responsible for websocket setup, 7286 including authentication, etc. 7287 7288 It also gets data sent to it by other processes and is responsible 7289 for distributing that, as necessary. 7290 */ 7291 void runWebsocketServer()() { 7292 assert(0, "not implemented"); 7293 } 7294 7295 void sendToWebsocketServer(WebSocket ws, string group) { 7296 assert(0, "not implemented"); 7297 } 7298 7299 void sendToWebsocketServer(string content, string group) { 7300 assert(0, "not implemented"); 7301 } 7302 7303 7304 void runEventServer()() { 7305 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 7306 } 7307 7308 void runTimerServer()() { 7309 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 7310 } 7311 7312 version(Posix) { 7313 alias LocalServerConnectionHandle = int; 7314 alias CgiConnectionHandle = int; 7315 alias SocketConnectionHandle = int; 7316 7317 enum INVALID_CGI_CONNECTION_HANDLE = -1; 7318 } else version(Windows) { 7319 alias LocalServerConnectionHandle = HANDLE; 7320 version(embedded_httpd_threads) { 7321 alias CgiConnectionHandle = SOCKET; 7322 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7323 } else version(fastcgi) { 7324 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 7325 enum INVALID_CGI_CONNECTION_HANDLE = null; 7326 } else version(scgi) { 7327 alias CgiConnectionHandle = SOCKET; 7328 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 7329 } else { /* version(plain_cgi) */ 7330 alias CgiConnectionHandle = HANDLE; 7331 enum INVALID_CGI_CONNECTION_HANDLE = null; 7332 } 7333 alias SocketConnectionHandle = SOCKET; 7334 } 7335 7336 version(with_addon_servers_connections) 7337 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 7338 version(Posix) { 7339 import core.sys.posix.unistd; 7340 import core.sys.posix.sys.un; 7341 7342 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 7343 if(sock == -1) 7344 throw new Exception("socket " ~ to!string(errno)); 7345 7346 scope(failure) 7347 close(sock); 7348 7349 cloexec(sock); 7350 7351 // add-on server processes are assumed to be local, and thus will 7352 // use unix domain sockets. Besides, I want to pass sockets to them, 7353 // so it basically must be local (except for the session server, but meh). 7354 sockaddr_un addr; 7355 addr.sun_family = AF_UNIX; 7356 version(linux) { 7357 // on linux, we will use the abstract namespace 7358 addr.sun_path[0] = 0; 7359 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 7360 } else { 7361 // but otherwise, just use a file cuz we must. 7362 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 7363 } 7364 7365 bool alreadyTried; 7366 7367 try_again: 7368 7369 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 7370 if(!alreadyTried && errno == ECONNREFUSED) { 7371 // try auto-spawning the server, then attempt connection again 7372 startAddonServer(arg); 7373 import core.thread; 7374 Thread.sleep(50.msecs); 7375 alreadyTried = true; 7376 goto try_again; 7377 } else 7378 throw new Exception("connect " ~ to!string(errno)); 7379 } 7380 7381 return sock; 7382 } else version(Windows) { 7383 return null; // FIXME 7384 } 7385 } 7386 7387 version(with_addon_servers_connections) 7388 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 7389 version(Posix) { 7390 import core.sys.posix.unistd; 7391 close(handle); 7392 } else version(Windows) 7393 CloseHandle(handle); 7394 } 7395 7396 void runSessionServer()() { 7397 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 7398 } 7399 7400 import core.stdc.errno; 7401 7402 struct IoOp { 7403 @disable this(); 7404 @disable this(this); 7405 7406 /* 7407 So we want to be able to eventually handle generic sockets too. 7408 */ 7409 7410 enum Read = 1; 7411 enum Write = 2; 7412 enum Accept = 3; 7413 enum ReadSocketHandle = 4; 7414 7415 // Your handler may be called in a different thread than the one that initiated the IO request! 7416 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 7417 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 7418 private void delegate(IoOp*) closeHandler; 7419 private void delegate(IoOp*) completeHandler; 7420 private int internalFd; 7421 private int operation; 7422 private int bufferLengthAllocated; 7423 private int bufferLengthUsed; 7424 private ubyte[1] internalBuffer; // it can be overallocated! 7425 7426 ubyte[] allocatedBuffer() return { 7427 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 7428 } 7429 7430 ubyte[] usedBuffer() return { 7431 return allocatedBuffer[0 .. bufferLengthUsed]; 7432 } 7433 7434 void reset() { 7435 bufferLengthUsed = 0; 7436 } 7437 7438 int fd() { 7439 return internalFd; 7440 } 7441 } 7442 7443 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 7444 import core.stdc.stdlib; 7445 7446 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 7447 if(ptr is null) 7448 assert(0); // out of memory! 7449 7450 auto op = cast(IoOp*) ptr; 7451 7452 op.handler = handler; 7453 op.internalFd = fd; 7454 op.operation = operation; 7455 op.bufferLengthAllocated = bufferSize; 7456 op.bufferLengthUsed = 0; 7457 7458 import core.memory; 7459 7460 GC.addRoot(ptr); 7461 7462 return op; 7463 } 7464 7465 void freeIoOp(ref IoOp* ptr) { 7466 7467 import core.memory; 7468 GC.removeRoot(ptr); 7469 7470 import core.stdc.stdlib; 7471 free(ptr); 7472 ptr = null; 7473 } 7474 7475 version(Posix) 7476 version(with_addon_servers_connections) 7477 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7478 7479 //import std.stdio : writeln; writeln(cast(string) data); 7480 7481 import core.sys.posix.unistd; 7482 7483 auto ret = write(connection, data.ptr, data.length); 7484 if(ret != data.length) { 7485 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 7486 // the file is closed, remove it 7487 eis.fileClosed(connection); 7488 } else 7489 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 7490 } 7491 } 7492 version(Windows) 7493 version(with_addon_servers_connections) 7494 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 7495 // FIXME 7496 } 7497 7498 bool isInvalidHandle(CgiConnectionHandle h) { 7499 return h == INVALID_CGI_CONNECTION_HANDLE; 7500 } 7501 7502 /+ 7503 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 7504 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 7505 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 7506 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 7507 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 7508 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 7509 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 7510 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 7511 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 7512 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 7513 7514 +/ 7515 7516 /++ 7517 You can customize your server by subclassing the appropriate server. Then, register your 7518 subclass at compile time with the [registerEventIoServer] template, or implement your own 7519 main function and call it yourself. 7520 7521 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7522 +/ 7523 version(with_addon_servers_connections) 7524 interface EventIoServer { 7525 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7526 void handleLocalConnectionClose(IoOp* op); 7527 void handleLocalConnectionComplete(IoOp* op); 7528 void wait_timeout(); 7529 void fileClosed(int fd); 7530 7531 void epoll_fd(int fd); 7532 } 7533 7534 // the sink should buffer it 7535 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7536 static if(is(T == struct)) { 7537 foreach(member; __traits(allMembers, T)) 7538 serialize(sink, __traits(getMember, t, member)); 7539 } else static if(is(T : int)) { 7540 // no need to think of endianness just because this is only used 7541 // for local, same-machine stuff anyway. thanks private lol 7542 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7543 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7544 // these are common enough to optimize 7545 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7546 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7547 sink(cast(ubyte[]) t[]); 7548 } else static if(is(T : A[], A)) { 7549 // generic array is less optimal but still prolly ok 7550 int len = cast(int) t.length; 7551 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7552 foreach(item; t) 7553 serialize(sink, item); 7554 } else static assert(0, T.stringof); 7555 } 7556 7557 // all may be stack buffers, so use cautio 7558 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7559 static if(is(T == struct)) { 7560 T t; 7561 foreach(member; __traits(allMembers, T)) 7562 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7563 dg(t); 7564 } else static if(is(T : int)) { 7565 // no need to think of endianness just because this is only used 7566 // for local, same-machine stuff anyway. thanks private lol 7567 T t; 7568 auto data = get(t.sizeof); 7569 t = (cast(T[]) data)[0]; 7570 dg(t); 7571 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7572 // these are common enough to optimize 7573 int len; 7574 auto data = get(len.sizeof); 7575 len = (cast(int[]) data)[0]; 7576 7577 /* 7578 typeof(T[0])[2000] stackBuffer; 7579 T buffer; 7580 7581 if(len < stackBuffer.length) 7582 buffer = stackBuffer[0 .. len]; 7583 else 7584 buffer = new T(len); 7585 7586 data = get(len * typeof(T[0]).sizeof); 7587 */ 7588 7589 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7590 7591 dg(t); 7592 } else static if(is(T == E[], E)) { 7593 T t; 7594 int len; 7595 auto data = get(len.sizeof); 7596 len = (cast(int[]) data)[0]; 7597 t.length = len; 7598 foreach(ref e; t) { 7599 deserialize!E(get, (ele) { e = ele; }); 7600 } 7601 dg(t); 7602 } else static assert(0, T.stringof); 7603 } 7604 7605 unittest { 7606 serialize((ubyte[] b) { 7607 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7608 }, 1); 7609 serialize((ubyte[] b) { 7610 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7611 }, 56674); 7612 ubyte[1000] buffer; 7613 int bufferPoint; 7614 void add(scope ubyte[] b) { 7615 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7616 bufferPoint += b.length; 7617 } 7618 ubyte[] get(int sz) { 7619 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7620 bufferPoint += sz; 7621 return b; 7622 } 7623 serialize(&add, "test here"); 7624 bufferPoint = 0; 7625 deserialize!string(&get, (t) { assert(t == "test here"); }); 7626 bufferPoint = 0; 7627 7628 struct Foo { 7629 int a; 7630 ubyte c; 7631 string d; 7632 } 7633 serialize(&add, Foo(403, 37, "amazing")); 7634 bufferPoint = 0; 7635 deserialize!Foo(&get, (t) { 7636 assert(t.a == 403); 7637 assert(t.c == 37); 7638 assert(t.d == "amazing"); 7639 }); 7640 bufferPoint = 0; 7641 } 7642 7643 /* 7644 Here's the way the RPC interface works: 7645 7646 You define the interface that lists the functions you can call on the remote process. 7647 The interface may also have static methods for convenience. These forward to a singleton 7648 instance of an auto-generated class, which actually sends the args over the pipe. 7649 7650 An impl class actually implements it. A receiving server deserializes down the pipe and 7651 calls methods on the class. 7652 7653 I went with the interface to get some nice compiler checking and documentation stuff. 7654 7655 I could have skipped the interface and just implemented it all from the server class definition 7656 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7657 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7658 7659 7660 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7661 7662 Realistically though the bodies would just be 7663 connection.call(this.mangleof, args...) sooooo. 7664 7665 FIXME: overloads aren't supported 7666 */ 7667 7668 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7669 interface SessionObject {} 7670 7671 private immutable void delegate(string[])[string] scheduledJobHandlers; 7672 private immutable void delegate(string[])[string] websocketServers; 7673 7674 version(with_breaking_cgi_features) 7675 mixin(q{ 7676 7677 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7678 static import std.traits; 7679 7680 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7681 static foreach(idx, member; __traits(derivedMembers, T)) { 7682 static if(__traits(isVirtualMethod, __traits(getMember, T, member))) 7683 mixin( q{ 7684 std.traits.ReturnType!(__traits(getMember, T, member)) 7685 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7686 { 7687 SerializationBuffer buffer; 7688 auto i = cast(ushort) idx; 7689 serialize(&buffer.sink, i); 7690 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7691 foreach(param; params) 7692 serialize(&buffer.sink, param); 7693 7694 auto sendable = buffer.sendable; 7695 7696 version(Posix) {{ 7697 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7698 7699 if(ret == -1) { 7700 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7701 } else if(ret == 0) { 7702 throw new Exception("Connection to addon server lost"); 7703 } if(ret < sendable.length) 7704 throw new Exception("Send failed to send all"); 7705 assert(ret == sendable.length); 7706 }} // FIXME Windows impl 7707 7708 static if(!is(typeof(return) == void)) { 7709 // there is a return value; we need to wait for it too 7710 version(Posix) { 7711 ubyte[3000] revBuffer; 7712 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7713 auto got = revBuffer[0 .. ret]; 7714 7715 int dataLocation; 7716 ubyte[] grab(int sz) { 7717 auto dataLocation1 = dataLocation; 7718 dataLocation += sz; 7719 return got[dataLocation1 .. dataLocation]; 7720 } 7721 7722 typeof(return) retu; 7723 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7724 return retu; 7725 } else { 7726 // FIXME Windows impl 7727 return typeof(return).init; 7728 } 7729 7730 } 7731 }}); 7732 } 7733 7734 private static typeof(this) singletonInstance; 7735 private LocalServerConnectionHandle connectionHandle; 7736 7737 static typeof(this) connection() { 7738 if(singletonInstance is null) { 7739 singletonInstance = new typeof(this)(); 7740 singletonInstance.connect(); 7741 } 7742 return singletonInstance; 7743 } 7744 7745 void connect() { 7746 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7747 } 7748 7749 void disconnect() { 7750 closeLocalServerConnection(connectionHandle); 7751 } 7752 } 7753 7754 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7755 ushort calledIdx; 7756 string calledFunction; 7757 7758 int dataLocation; 7759 ubyte[] grab(int sz) { 7760 if(sz == 0) assert(0); 7761 auto d = data[dataLocation .. dataLocation + sz]; 7762 dataLocation += sz; 7763 return d; 7764 } 7765 7766 again: 7767 7768 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7769 deserialize!string(&grab, (a) { calledFunction = a; }); 7770 7771 import std.traits; 7772 7773 sw: switch(calledIdx) { 7774 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7775 static if(__traits(isVirtualMethod, __traits(getMember, Interface, memberName))) { 7776 case idx: 7777 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7778 7779 Parameters!(__traits(getMember, Interface, memberName)) params; 7780 foreach(ref param; params) 7781 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7782 7783 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7784 __traits(getMember, this_, memberName)(params); 7785 } else { 7786 auto ret = __traits(getMember, this_, memberName)(params); 7787 SerializationBuffer buffer; 7788 serialize(&buffer.sink, ret); 7789 7790 auto sendable = buffer.sendable; 7791 7792 version(Posix) { 7793 auto r = send(fd, sendable.ptr, sendable.length, 0); 7794 if(r == -1) { 7795 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7796 } else if(r == 0) { 7797 throw new Exception("Connection to addon client lost"); 7798 } if(r < sendable.length) 7799 throw new Exception("Send failed to send all"); 7800 7801 } // FIXME Windows impl 7802 } 7803 break sw; 7804 } 7805 default: assert(0); 7806 } 7807 7808 if(dataLocation != data.length) 7809 goto again; 7810 } 7811 7812 7813 private struct SerializationBuffer { 7814 ubyte[2048] bufferBacking; 7815 int bufferLocation; 7816 void sink(scope ubyte[] data) { 7817 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7818 bufferLocation += data.length; 7819 } 7820 7821 ubyte[] sendable() return { 7822 return bufferBacking[0 .. bufferLocation]; 7823 } 7824 } 7825 7826 /* 7827 FIXME: 7828 add a version command line arg 7829 version data in the library 7830 management gui as external program 7831 7832 at server with event_fd for each run 7833 use .mangleof in the at function name 7834 7835 i think the at server will have to: 7836 pipe args to the child 7837 collect child output for logging 7838 get child return value for logging 7839 7840 on windows timers work differently. idk how to best combine with the io stuff. 7841 7842 will have to have dump and restore too, so i can restart without losing stuff. 7843 */ 7844 7845 /++ 7846 A convenience object for talking to the [BasicDataServer] from a higher level. 7847 See: [Cgi.getSessionObject]. 7848 7849 You pass it a `Data` struct describing the data you want saved in the session. 7850 Then, this class will generate getter and setter properties that allow access 7851 to that data. 7852 7853 Note that each load and store will be done as-accessed; it doesn't front-load 7854 mutable data nor does it batch updates out of fear of read-modify-write race 7855 conditions. (In fact, right now it does this for everything, but in the future, 7856 I might batch load `immutable` members of the Data struct.) 7857 7858 At some point in the future, I might also let it do different backends, like 7859 a client-side cookie store too, but idk. 7860 7861 Note that the plain-old-data members of your `Data` struct are wrapped by this 7862 interface via a static foreach to make property functions. 7863 7864 See_Also: [MockSession] 7865 +/ 7866 interface Session(Data) : SessionObject { 7867 @property string sessionId() const; 7868 7869 /++ 7870 Starts a new session. Note that a session is also 7871 implicitly started as soon as you write data to it, 7872 so if you need to alter these parameters from their 7873 defaults, be sure to explicitly call this BEFORE doing 7874 any writes to session data. 7875 7876 Params: 7877 idleLifetime = How long, in seconds, the session 7878 should remain in memory when not being read from 7879 or written to. The default is one day. 7880 7881 NOT IMPLEMENTED 7882 7883 useExtendedLifetimeCookie = The session ID is always 7884 stored in a HTTP cookie, and by default, that cookie 7885 is discarded when the user closes their browser. 7886 7887 But if you set this to true, it will use a non-perishable 7888 cookie for the given idleLifetime. 7889 7890 NOT IMPLEMENTED 7891 +/ 7892 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7893 7894 /++ 7895 Regenerates the session ID and updates the associated 7896 cookie. 7897 7898 This is also your chance to change immutable data 7899 (not yet implemented). 7900 +/ 7901 void regenerateId(); 7902 7903 /++ 7904 Terminates this session, deleting all saved data. 7905 +/ 7906 void terminate(); 7907 7908 /++ 7909 Plain-old-data members of your `Data` struct are wrapped here via 7910 the property getters and setters. 7911 7912 If the member is a non-string array, it returns a magical array proxy 7913 object which allows for atomic appends and replaces via overloaded operators. 7914 You can slice this to get a range representing a $(B const) view of the array. 7915 This is to protect you against read-modify-write race conditions. 7916 +/ 7917 static foreach(memberName; __traits(allMembers, Data)) 7918 static if(is(typeof(__traits(getMember, Data, memberName)))) 7919 mixin(q{ 7920 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 7921 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 7922 }); 7923 7924 } 7925 7926 /++ 7927 An implementation of [Session] that works on real cgi connections utilizing the 7928 [BasicDataServer]. 7929 7930 As opposed to a [MockSession] which is made for testing purposes. 7931 7932 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 7933 +/ 7934 class BasicDataServerSession(Data) : Session!Data { 7935 private Cgi cgi; 7936 private string sessionId_; 7937 7938 public @property string sessionId() const { 7939 return sessionId_; 7940 } 7941 7942 protected @property string sessionId(string s) { 7943 return this.sessionId_ = s; 7944 } 7945 7946 private this(Cgi cgi) { 7947 this.cgi = cgi; 7948 if(auto ptr = "sessionId" in cgi.cookies) 7949 sessionId = (*ptr).length ? *ptr : null; 7950 } 7951 7952 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 7953 assert(sessionId is null); 7954 7955 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 7956 7957 import std.random, std.conv; 7958 sessionId = to!string(uniform(1, long.max)); 7959 7960 BasicDataServer.connection.createSession(sessionId, idleLifetime); 7961 setCookie(); 7962 } 7963 7964 protected void setCookie() { 7965 cgi.setCookie( 7966 "sessionId", sessionId, 7967 0 /* expiration */, 7968 "/" /* path */, 7969 null /* domain */, 7970 true /* http only */, 7971 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 7972 } 7973 7974 void regenerateId() { 7975 if(sessionId is null) { 7976 start(); 7977 return; 7978 } 7979 import std.random, std.conv; 7980 auto oldSessionId = sessionId; 7981 sessionId = to!string(uniform(1, long.max)); 7982 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 7983 setCookie(); 7984 } 7985 7986 void terminate() { 7987 BasicDataServer.connection.destroySession(sessionId); 7988 sessionId = null; 7989 setCookie(); 7990 } 7991 7992 static foreach(memberName; __traits(allMembers, Data)) 7993 static if(is(typeof(__traits(getMember, Data, memberName)))) 7994 mixin(q{ 7995 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7996 if(sessionId is null) 7997 return typeof(return).init; 7998 7999 import std.traits; 8000 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 8001 if(v.length == 0) 8002 return typeof(return).init; 8003 import std.conv; 8004 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 8005 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 8006 // basically. Assuming the session is POD this should be fine. 8007 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 8008 } 8009 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8010 if(sessionId is null) 8011 start(); 8012 import std.conv; 8013 import std.traits; 8014 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 8015 return value; 8016 } 8017 }); 8018 } 8019 8020 /++ 8021 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 8022 Simply stores the data in its instance members. 8023 +/ 8024 class MockSession(Data) : Session!Data { 8025 pure { 8026 @property string sessionId() const { return "mock"; } 8027 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 8028 void regenerateId() {} 8029 void terminate() {} 8030 8031 private Data store_; 8032 8033 static foreach(memberName; __traits(allMembers, Data)) 8034 static if(is(typeof(__traits(getMember, Data, memberName)))) 8035 mixin(q{ 8036 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 8037 return __traits(getMember, store_, memberName); 8038 } 8039 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 8040 return __traits(getMember, store_, memberName) = value; 8041 } 8042 }); 8043 } 8044 } 8045 8046 /++ 8047 Direct interface to the basic data add-on server. You can 8048 typically use [Cgi.getSessionObject] as a more convenient interface. 8049 +/ 8050 version(with_addon_servers_connections) 8051 interface BasicDataServer { 8052 /// 8053 void createSession(string sessionId, int lifetime); 8054 /// 8055 void renewSession(string sessionId, int lifetime); 8056 /// 8057 void destroySession(string sessionId); 8058 /// 8059 void renameSession(string oldSessionId, string newSessionId); 8060 8061 /// 8062 void setSessionData(string sessionId, string dataKey, string dataValue); 8063 /// 8064 string getSessionData(string sessionId, string dataKey); 8065 8066 /// 8067 static BasicDataServerConnection connection() { 8068 return BasicDataServerConnection.connection(); 8069 } 8070 } 8071 8072 version(with_addon_servers_connections) 8073 class BasicDataServerConnection : BasicDataServer { 8074 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 8075 } 8076 8077 version(with_addon_servers) 8078 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 8079 8080 void createSession(string sessionId, int lifetime) { 8081 sessions[sessionId.idup] = Session(lifetime); 8082 } 8083 void destroySession(string sessionId) { 8084 sessions.remove(sessionId); 8085 } 8086 void renewSession(string sessionId, int lifetime) { 8087 sessions[sessionId].lifetime = lifetime; 8088 } 8089 void renameSession(string oldSessionId, string newSessionId) { 8090 sessions[newSessionId.idup] = sessions[oldSessionId]; 8091 sessions.remove(oldSessionId); 8092 } 8093 void setSessionData(string sessionId, string dataKey, string dataValue) { 8094 if(sessionId !in sessions) 8095 createSession(sessionId, 3600); // FIXME? 8096 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 8097 } 8098 string getSessionData(string sessionId, string dataKey) { 8099 if(auto session = sessionId in sessions) { 8100 if(auto data = dataKey in (*session).values) 8101 return *data; 8102 else 8103 return null; // no such data 8104 8105 } else { 8106 return null; // no session 8107 } 8108 } 8109 8110 8111 protected: 8112 8113 struct Session { 8114 int lifetime; 8115 8116 string[string] values; 8117 } 8118 8119 Session[string] sessions; 8120 8121 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8122 auto data = op.usedBuffer; 8123 dispatchRpcServer!BasicDataServer(this, data, op.fd); 8124 return false; 8125 } 8126 8127 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8128 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8129 void wait_timeout() {} 8130 void fileClosed(int fd) {} // stateless so irrelevant 8131 void epoll_fd(int fd) {} 8132 } 8133 8134 /++ 8135 See [schedule] to make one of these. You then call one of the methods here to set it up: 8136 8137 --- 8138 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 8139 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 8140 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 8141 --- 8142 +/ 8143 version(with_addon_servers_connections) 8144 struct ScheduledJobHelper { 8145 private string func; 8146 private string[] args; 8147 private bool consumed; 8148 8149 private this(string func, string[] args) { 8150 this.func = func; 8151 this.args = args; 8152 } 8153 8154 ~this() { 8155 assert(consumed); 8156 } 8157 8158 /++ 8159 Schedules the job to be run at the given time. 8160 +/ 8161 void at(DateTime when, immutable TimeZone timezone = UTC()) { 8162 consumed = true; 8163 8164 auto conn = ScheduledJobServerConnection.connection; 8165 import std.file; 8166 auto st = SysTime(when, timezone); 8167 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 8168 } 8169 8170 /++ 8171 Schedules the job to run at least after the specified delay. 8172 +/ 8173 void delay(Duration delay) { 8174 consumed = true; 8175 8176 auto conn = ScheduledJobServerConnection.connection; 8177 import std.file; 8178 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 8179 } 8180 8181 /++ 8182 Runs the job in the background ASAP. 8183 8184 $(NOTE It may run in a background thread. Don't segfault!) 8185 +/ 8186 void asap() { 8187 consumed = true; 8188 8189 auto conn = ScheduledJobServerConnection.connection; 8190 import std.file; 8191 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 8192 } 8193 8194 /+ 8195 /++ 8196 Schedules the job to recur on the given pattern. 8197 +/ 8198 void recur(string spec) { 8199 8200 } 8201 +/ 8202 } 8203 8204 /++ 8205 First step to schedule a job on the scheduled job server. 8206 8207 The scheduled job needs to be a top-level function that doesn't read any 8208 variables from outside its arguments because it may be run in a new process, 8209 without any context existing later. 8210 8211 You MUST set details on the returned object to actually do anything! 8212 +/ 8213 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 8214 /// 8215 ScheduledJobHelper schedule(T args) { 8216 // this isn't meant to ever be called, but instead just to 8217 // get the compiler to type check the arguments passed for us 8218 auto sample = delegate() { 8219 fn(args); 8220 }; 8221 string[] sargs; 8222 foreach(arg; args) 8223 sargs ~= to!string(arg); 8224 return ScheduledJobHelper(fn.mangleof, sargs); 8225 } 8226 8227 shared static this() { 8228 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 8229 import std.traits; 8230 Parameters!fn args; 8231 foreach(idx, ref arg; args) 8232 arg = to!(typeof(arg))(sargs[idx]); 8233 fn(args); 8234 }; 8235 } 8236 } 8237 8238 /// 8239 interface ScheduledJobServer { 8240 /// Use the [schedule] function for a higher-level interface. 8241 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 8242 /// 8243 void cancelJob(int jobId); 8244 } 8245 8246 version(with_addon_servers_connections) 8247 class ScheduledJobServerConnection : ScheduledJobServer { 8248 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 8249 } 8250 8251 version(with_addon_servers) 8252 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 8253 // FIXME: we need to handle SIGCHLD in this somehow 8254 // whenIs is 0 for relative, 1 for absolute 8255 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 8256 auto nj = nextJobId; 8257 nextJobId++; 8258 8259 version(linux) { 8260 import core.sys.linux.timerfd; 8261 import core.sys.linux.epoll; 8262 import core.sys.posix.unistd; 8263 8264 8265 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 8266 if(fd == -1) 8267 throw new Exception("fd timer create failed"); 8268 8269 foreach(ref arg; args) 8270 arg = arg.idup; 8271 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 8272 8273 itimerspec value; 8274 value.it_value.tv_sec = when; 8275 value.it_value.tv_nsec = 0; 8276 8277 value.it_interval.tv_sec = 0; 8278 value.it_interval.tv_nsec = 0; 8279 8280 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 8281 throw new Exception("couldn't set fd timer"); 8282 8283 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 8284 jobs.remove(nj); 8285 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 8286 close(fd); 8287 8288 8289 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 8290 8291 return true; 8292 }); 8293 scope(failure) 8294 freeIoOp(op); 8295 8296 epoll_event ev; 8297 ev.events = EPOLLIN | EPOLLET; 8298 ev.data.ptr = op; 8299 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 8300 throw new Exception("epoll_ctl " ~ to!string(errno)); 8301 8302 jobs[nj] = job; 8303 return nj; 8304 } else assert(0); 8305 } 8306 8307 protected void cancelJob(int jobId) { 8308 version(linux) { 8309 auto job = jobId in jobs; 8310 if(job is null) 8311 return; 8312 8313 jobs.remove(jobId); 8314 8315 version(linux) { 8316 import core.sys.linux.timerfd; 8317 import core.sys.linux.epoll; 8318 import core.sys.posix.unistd; 8319 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 8320 close(job.timerfd); 8321 } 8322 } 8323 jobs.remove(jobId); 8324 } 8325 8326 int nextJobId = 1; 8327 static struct Job { 8328 string executable; 8329 string func; 8330 string[] args; 8331 int timerfd; 8332 int id; 8333 } 8334 Job[int] jobs; 8335 8336 8337 // event io server methods below 8338 8339 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8340 auto data = op.usedBuffer; 8341 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 8342 return false; 8343 } 8344 8345 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 8346 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 8347 void wait_timeout() {} 8348 void fileClosed(int fd) {} // stateless so irrelevant 8349 8350 int epoll_fd_; 8351 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 8352 int epoll_fd() { return epoll_fd_; } 8353 } 8354 8355 /// 8356 version(with_addon_servers_connections) 8357 interface EventSourceServer { 8358 /++ 8359 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 8360 8361 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8362 8363 See_Also: 8364 [sendEvent] 8365 +/ 8366 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 8367 /* 8368 If lastEventId is missing or empty, you just get new events as they come. 8369 8370 If it is set from something else, it sends all since then (that are still alive) 8371 down the pipe immediately. 8372 8373 The reason it can come from the header is that's what the standard defines for 8374 browser reconnects. The reason it can come from a query string is just convenience 8375 in catching up in a user-defined manner. 8376 8377 The reason the header overrides the query string is if the browser tries to reconnect, 8378 it will send the header AND the query (it reconnects to the same url), so we just 8379 want to do the restart thing. 8380 8381 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 8382 */ 8383 string lastEventId = cgi.lastEventId; 8384 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 8385 lastEventId = cgi.get["lastEventId"]; 8386 8387 cgi.setResponseContentType("text/event-stream"); 8388 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 8389 cgi.flush(); 8390 8391 cgi.closed = true; 8392 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8393 scope(exit) 8394 closeLocalServerConnection(s); 8395 8396 version(fastcgi) 8397 throw new Exception("sending fcgi connections not supported"); 8398 else { 8399 auto fd = cgi.getOutputFileHandle(); 8400 if(isInvalidHandle(fd)) 8401 throw new Exception("bad fd from cgi!"); 8402 8403 EventSourceServerImplementation.SendableEventConnection sec; 8404 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 8405 8406 version(Posix) { 8407 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 8408 assert(res == sec.sizeof); 8409 } else version(Windows) { 8410 // FIXME 8411 } 8412 } 8413 } 8414 8415 /++ 8416 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 8417 8418 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 8419 8420 Params: 8421 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 8422 event = the event type string, which is used in the Javascript addEventListener API on EventSource 8423 data = the event data. Available in JS as `event.data`. 8424 lifetime = the amount of time to keep this event for replaying on the event server. 8425 8426 See_Also: 8427 [sendEventToEventServer] 8428 +/ 8429 public static void sendEvent(string url, string event, string data, int lifetime) { 8430 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 8431 scope(exit) 8432 closeLocalServerConnection(s); 8433 8434 EventSourceServerImplementation.SendableEvent sev; 8435 sev.populate(url, event, data, lifetime); 8436 8437 version(Posix) { 8438 auto ret = send(s, &sev, sev.sizeof, 0); 8439 assert(ret == sev.sizeof); 8440 } else version(Windows) { 8441 // FIXME 8442 } 8443 } 8444 8445 /++ 8446 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 8447 8448 See_Also: [disconnect] 8449 +/ 8450 void connect(string url, string forwardUrl); 8451 8452 /++ 8453 Disconnects `forwardUrl` from `url` 8454 8455 See_Also: [connect] 8456 +/ 8457 void disconnect(string url, string forwardUrl); 8458 } 8459 8460 /// 8461 version(with_addon_servers) 8462 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 8463 8464 protected: 8465 8466 void connect(string url, string forwardUrl) { 8467 pipes[url] ~= forwardUrl; 8468 } 8469 void disconnect(string url, string forwardUrl) { 8470 auto t = url in pipes; 8471 if(t is null) 8472 return; 8473 foreach(idx, n; (*t)) 8474 if(n == forwardUrl) { 8475 (*t)[idx] = (*t)[$-1]; 8476 (*t) = (*t)[0 .. $-1]; 8477 break; 8478 } 8479 } 8480 8481 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 8482 if(receivedFd != -1) { 8483 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 8484 8485 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 8486 8487 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 8488 8489 auto url = got.url.idup; 8490 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 8491 8492 // FIXME: catch up on past messages here 8493 } else { 8494 auto data = op.usedBuffer; 8495 auto event = cast(SendableEvent*) data.ptr; 8496 8497 if(event.magic == 0xdeadbeef) { 8498 handleInputEvent(event); 8499 8500 if(event.url in pipes) 8501 foreach(pipe; pipes[event.url]) { 8502 event.url = pipe; 8503 handleInputEvent(event); 8504 } 8505 } else { 8506 dispatchRpcServer!EventSourceServer(this, data, op.fd); 8507 } 8508 } 8509 return false; 8510 } 8511 void handleLocalConnectionClose(IoOp* op) { 8512 fileClosed(op.fd); 8513 } 8514 void handleLocalConnectionComplete(IoOp* op) {} 8515 8516 void wait_timeout() { 8517 // just keeping alive 8518 foreach(url, connections; eventConnectionsByUrl) 8519 foreach(connection; connections) 8520 if(connection.needsChunking) 8521 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8522 else 8523 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8524 } 8525 8526 void fileClosed(int fd) { 8527 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8528 foreach(idx, conn; connections) { 8529 if(fd == conn.fd) { 8530 connections[idx] = connections[$-1]; 8531 connections = connections[0 .. $ - 1]; 8532 continue outer; 8533 } 8534 } 8535 } 8536 } 8537 8538 void epoll_fd(int fd) {} 8539 8540 8541 private: 8542 8543 8544 struct SendableEventConnection { 8545 ubyte responseChunked; 8546 8547 int urlLength; 8548 char[256] urlBuffer = 0; 8549 8550 int lastEventIdLength; 8551 char[32] lastEventIdBuffer = 0; 8552 8553 char[] url() return { 8554 return urlBuffer[0 .. urlLength]; 8555 } 8556 void url(in char[] u) { 8557 urlBuffer[0 .. u.length] = u[]; 8558 urlLength = cast(int) u.length; 8559 } 8560 char[] lastEventId() return { 8561 return lastEventIdBuffer[0 .. lastEventIdLength]; 8562 } 8563 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8564 in { 8565 assert(url.length < this.urlBuffer.length); 8566 assert(lastEventId.length < this.lastEventIdBuffer.length); 8567 } 8568 do { 8569 this.responseChunked = responseChunked ? 1 : 0; 8570 this.urlLength = cast(int) url.length; 8571 this.lastEventIdLength = cast(int) lastEventId.length; 8572 8573 this.urlBuffer[0 .. url.length] = url[]; 8574 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8575 } 8576 } 8577 8578 struct SendableEvent { 8579 int magic = 0xdeadbeef; 8580 int urlLength; 8581 char[256] urlBuffer = 0; 8582 int typeLength; 8583 char[32] typeBuffer = 0; 8584 int messageLength; 8585 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8586 int _lifetime; 8587 8588 char[] message() return { 8589 return messageBuffer[0 .. messageLength]; 8590 } 8591 char[] type() return { 8592 return typeBuffer[0 .. typeLength]; 8593 } 8594 char[] url() return { 8595 return urlBuffer[0 .. urlLength]; 8596 } 8597 void url(in char[] u) { 8598 urlBuffer[0 .. u.length] = u[]; 8599 urlLength = cast(int) u.length; 8600 } 8601 int lifetime() { 8602 return _lifetime; 8603 } 8604 8605 /// 8606 void populate(string url, string type, string message, int lifetime) 8607 in { 8608 assert(url.length < this.urlBuffer.length); 8609 assert(type.length < this.typeBuffer.length); 8610 assert(message.length < this.messageBuffer.length); 8611 } 8612 do { 8613 this.urlLength = cast(int) url.length; 8614 this.typeLength = cast(int) type.length; 8615 this.messageLength = cast(int) message.length; 8616 this._lifetime = lifetime; 8617 8618 this.urlBuffer[0 .. url.length] = url[]; 8619 this.typeBuffer[0 .. type.length] = type[]; 8620 this.messageBuffer[0 .. message.length] = message[]; 8621 } 8622 } 8623 8624 struct EventConnection { 8625 int fd; 8626 bool needsChunking; 8627 } 8628 8629 private EventConnection[][string] eventConnectionsByUrl; 8630 private string[][string] pipes; 8631 8632 private void handleInputEvent(scope SendableEvent* event) { 8633 static int eventId; 8634 8635 static struct StoredEvent { 8636 int id; 8637 string type; 8638 string message; 8639 int lifetimeRemaining; 8640 } 8641 8642 StoredEvent[][string] byUrl; 8643 8644 int thisId = ++eventId; 8645 8646 if(event.lifetime) 8647 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8648 8649 auto connectionsPtr = event.url in eventConnectionsByUrl; 8650 EventConnection[] connections; 8651 if(connectionsPtr is null) 8652 return; 8653 else 8654 connections = *connectionsPtr; 8655 8656 char[4096] buffer; 8657 char[] formattedMessage; 8658 8659 void append(const char[] a) { 8660 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8661 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8662 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8663 } 8664 8665 import std.algorithm.iteration; 8666 8667 if(connections.length) { 8668 append("id: "); 8669 append(to!string(thisId)); 8670 append("\n"); 8671 8672 append("event: "); 8673 append(event.type); 8674 append("\n"); 8675 8676 foreach(line; event.message.splitter("\n")) { 8677 append("data: "); 8678 append(line); 8679 append("\n"); 8680 } 8681 8682 append("\n"); 8683 } 8684 8685 // chunk it for HTTP! 8686 auto len = toHex(formattedMessage.length); 8687 buffer[4 .. 6] = "\r\n"[]; 8688 buffer[4 - len.length .. 4] = len[]; 8689 buffer[6 + formattedMessage.length] = '\r'; 8690 buffer[6 + formattedMessage.length + 1] = '\n'; 8691 8692 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8693 // done 8694 8695 // FIXME: send back requests when needed 8696 // FIXME: send a single ":\n" every 15 seconds to keep alive 8697 8698 foreach(connection; connections) { 8699 if(connection.needsChunking) { 8700 nonBlockingWrite(this, connection.fd, chunkedMessage); 8701 } else { 8702 nonBlockingWrite(this, connection.fd, formattedMessage); 8703 } 8704 } 8705 } 8706 } 8707 8708 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8709 version(Posix) { 8710 8711 import core.sys.posix.unistd; 8712 import core.sys.posix.fcntl; 8713 import core.sys.posix.sys.un; 8714 8715 import core.sys.posix.signal; 8716 signal(SIGPIPE, SIG_IGN); 8717 8718 static extern(C) void sigchldhandler(int) { 8719 int status; 8720 import w = core.sys.posix.sys.wait; 8721 w.wait(&status); 8722 } 8723 signal(SIGCHLD, &sigchldhandler); 8724 8725 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8726 if(sock == -1) 8727 throw new Exception("socket " ~ to!string(errno)); 8728 8729 scope(failure) 8730 close(sock); 8731 8732 cloexec(sock); 8733 8734 // add-on server processes are assumed to be local, and thus will 8735 // use unix domain sockets. Besides, I want to pass sockets to them, 8736 // so it basically must be local (except for the session server, but meh). 8737 sockaddr_un addr; 8738 addr.sun_family = AF_UNIX; 8739 version(linux) { 8740 // on linux, we will use the abstract namespace 8741 addr.sun_path[0] = 0; 8742 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8743 } else { 8744 // but otherwise, just use a file cuz we must. 8745 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8746 } 8747 8748 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8749 throw new Exception("bind " ~ to!string(errno)); 8750 8751 if(listen(sock, 128) == -1) 8752 throw new Exception("listen " ~ to!string(errno)); 8753 8754 makeNonBlocking(sock); 8755 8756 version(linux) { 8757 import core.sys.linux.epoll; 8758 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8759 if(epoll_fd == -1) 8760 throw new Exception("epoll_create1 " ~ to!string(errno)); 8761 scope(failure) 8762 close(epoll_fd); 8763 } else { 8764 import core.sys.posix.poll; 8765 } 8766 8767 version(linux) 8768 eis.epoll_fd = epoll_fd; 8769 8770 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8771 scope(exit) 8772 freeIoOp(acceptOp); 8773 8774 version(linux) { 8775 epoll_event ev; 8776 ev.events = EPOLLIN | EPOLLET; 8777 ev.data.ptr = acceptOp; 8778 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8779 throw new Exception("epoll_ctl " ~ to!string(errno)); 8780 8781 epoll_event[64] events; 8782 } else { 8783 pollfd[] pollfds; 8784 IoOp*[int] ioops; 8785 pollfds ~= pollfd(sock, POLLIN); 8786 ioops[sock] = acceptOp; 8787 } 8788 8789 import core.time : MonoTime, seconds; 8790 8791 MonoTime timeout = MonoTime.currTime + 15.seconds; 8792 8793 while(true) { 8794 8795 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8796 8797 int timeout_milliseconds = 0; // -1; // infinite 8798 8799 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8800 if(timeout_milliseconds < 0) 8801 timeout_milliseconds = 0; 8802 8803 //writeln("waiting for ", name); 8804 8805 version(linux) { 8806 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8807 if(nfds == -1) { 8808 if(errno == EINTR) 8809 continue; 8810 throw new Exception("epoll_wait " ~ to!string(errno)); 8811 } 8812 } else { 8813 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8814 size_t lastIdx = 0; 8815 } 8816 8817 if(nfds == 0) { 8818 eis.wait_timeout(); 8819 timeout += 15.seconds; 8820 } 8821 8822 foreach(idx; 0 .. nfds) { 8823 version(linux) { 8824 auto flags = events[idx].events; 8825 auto ioop = cast(IoOp*) events[idx].data.ptr; 8826 } else { 8827 IoOp* ioop; 8828 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8829 if(thing.revents) { 8830 ioop = ioops[thing.fd]; 8831 lastIdx += tidx + 1; 8832 break; 8833 } 8834 } 8835 } 8836 8837 //writeln(flags, " ", ioop.fd); 8838 8839 void newConnection() { 8840 // on edge triggering, it is important that we get it all 8841 while(true) { 8842 auto size = cast(socklen_t) addr.sizeof; 8843 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8844 if(ns == -1) { 8845 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8846 // all done, got it all 8847 break; 8848 } 8849 throw new Exception("accept " ~ to!string(errno)); 8850 } 8851 cloexec(ns); 8852 8853 makeNonBlocking(ns); 8854 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8855 niop.closeHandler = &eis.handleLocalConnectionClose; 8856 niop.completeHandler = &eis.handleLocalConnectionComplete; 8857 scope(failure) freeIoOp(niop); 8858 8859 version(linux) { 8860 epoll_event nev; 8861 nev.events = EPOLLIN | EPOLLET; 8862 nev.data.ptr = niop; 8863 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8864 throw new Exception("epoll_ctl " ~ to!string(errno)); 8865 } else { 8866 bool found = false; 8867 foreach(ref pfd; pollfds) { 8868 if(pfd.fd < 0) { 8869 pfd.fd = ns; 8870 found = true; 8871 } 8872 } 8873 if(!found) 8874 pollfds ~= pollfd(ns, POLLIN); 8875 ioops[ns] = niop; 8876 } 8877 } 8878 } 8879 8880 bool newConnectionCondition() { 8881 version(linux) 8882 return ioop.fd == sock && (flags & EPOLLIN); 8883 else 8884 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8885 } 8886 8887 if(newConnectionCondition()) { 8888 newConnection(); 8889 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8890 while(true) { 8891 int in_fd; 8892 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8893 if(got == -1) { 8894 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8895 // all done, got it all 8896 if(ioop.completeHandler) 8897 ioop.completeHandler(ioop); 8898 break; 8899 } 8900 throw new Exception("recv " ~ to!string(errno)); 8901 } 8902 8903 if(got == 0) { 8904 if(ioop.closeHandler) { 8905 ioop.closeHandler(ioop); 8906 version(linux) {} // nothing needed 8907 else { 8908 foreach(ref pfd; pollfds) { 8909 if(pfd.fd == ioop.fd) 8910 pfd.fd = -1; 8911 } 8912 } 8913 } 8914 close(ioop.fd); 8915 freeIoOp(ioop); 8916 break; 8917 } 8918 8919 ioop.bufferLengthUsed = cast(int) got; 8920 ioop.handler(ioop, in_fd); 8921 } 8922 } else if(ioop.operation == IoOp.Read) { 8923 while(true) { 8924 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 8925 if(got == -1) { 8926 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8927 // all done, got it all 8928 if(ioop.completeHandler) 8929 ioop.completeHandler(ioop); 8930 break; 8931 } 8932 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 8933 } 8934 8935 if(got == 0) { 8936 if(ioop.closeHandler) 8937 ioop.closeHandler(ioop); 8938 close(ioop.fd); 8939 freeIoOp(ioop); 8940 break; 8941 } 8942 8943 ioop.bufferLengthUsed = cast(int) got; 8944 if(ioop.handler(ioop, ioop.fd)) { 8945 close(ioop.fd); 8946 freeIoOp(ioop); 8947 break; 8948 } 8949 } 8950 } 8951 8952 // EPOLLHUP? 8953 } 8954 } 8955 } else version(Windows) { 8956 8957 // set up a named pipe 8958 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 8959 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 8960 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 8961 8962 } else static assert(0); 8963 } 8964 8965 8966 version(with_sendfd) 8967 // copied from the web and ported from C 8968 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 8969 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 8970 msghdr msg; 8971 iovec[1] iov; 8972 8973 version(OSX) { 8974 //msg.msg_accrights = cast(cattr_t) &sendfd; 8975 //msg.msg_accrightslen = int.sizeof; 8976 } else version(Android) { 8977 } else { 8978 union ControlUnion { 8979 cmsghdr cm; 8980 char[CMSG_SPACE(int.sizeof)] control; 8981 } 8982 8983 ControlUnion control_un; 8984 cmsghdr* cmptr; 8985 8986 msg.msg_control = control_un.control.ptr; 8987 msg.msg_controllen = control_un.control.length; 8988 8989 cmptr = CMSG_FIRSTHDR(&msg); 8990 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 8991 cmptr.cmsg_level = SOL_SOCKET; 8992 cmptr.cmsg_type = SCM_RIGHTS; 8993 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 8994 } 8995 8996 msg.msg_name = null; 8997 msg.msg_namelen = 0; 8998 8999 iov[0].iov_base = ptr; 9000 iov[0].iov_len = nbytes; 9001 msg.msg_iov = iov.ptr; 9002 msg.msg_iovlen = 1; 9003 9004 return sendmsg(fd, &msg, 0); 9005 } 9006 9007 version(with_sendfd) 9008 // copied from the web and ported from C 9009 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 9010 msghdr msg; 9011 iovec[1] iov; 9012 ssize_t n; 9013 int newfd; 9014 9015 version(OSX) { 9016 //msg.msg_accrights = cast(cattr_t) recvfd; 9017 //msg.msg_accrightslen = int.sizeof; 9018 } else version(Android) { 9019 } else { 9020 union ControlUnion { 9021 cmsghdr cm; 9022 char[CMSG_SPACE(int.sizeof)] control; 9023 } 9024 ControlUnion control_un; 9025 cmsghdr* cmptr; 9026 9027 msg.msg_control = control_un.control.ptr; 9028 msg.msg_controllen = control_un.control.length; 9029 } 9030 9031 msg.msg_name = null; 9032 msg.msg_namelen = 0; 9033 9034 iov[0].iov_base = ptr; 9035 iov[0].iov_len = nbytes; 9036 msg.msg_iov = iov.ptr; 9037 msg.msg_iovlen = 1; 9038 9039 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 9040 return n; 9041 9042 version(OSX) { 9043 //if(msg.msg_accrightslen != int.sizeof) 9044 //*recvfd = -1; 9045 } else version(Android) { 9046 } else { 9047 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 9048 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 9049 if (cmptr.cmsg_level != SOL_SOCKET) 9050 throw new Exception("control level != SOL_SOCKET"); 9051 if (cmptr.cmsg_type != SCM_RIGHTS) 9052 throw new Exception("control type != SCM_RIGHTS"); 9053 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 9054 } else 9055 *recvfd = -1; /* descriptor was not passed */ 9056 } 9057 9058 return n; 9059 } 9060 /* end read_fd */ 9061 9062 9063 /* 9064 Event source stuff 9065 9066 The api is: 9067 9068 sendEvent(string url, string type, string data, int timeout = 60*10); 9069 9070 attachEventListener(string url, int fd, lastId) 9071 9072 9073 It just sends to all attached listeners, and stores it until the timeout 9074 for replaying via lastEventId. 9075 */ 9076 9077 /* 9078 Session process stuff 9079 9080 it stores it all. the cgi object has a session object that can grab it 9081 9082 session may be done in the same process if possible, there is a version 9083 switch to choose if you want to override. 9084 */ 9085 9086 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 9087 alias handler = dispatchHandler; 9088 string urlPrefix; 9089 bool rejectFurther; 9090 immutable(DispatcherDetails) details; 9091 } 9092 9093 private string urlify(string name) pure { 9094 return beautify(name, '-', true); 9095 } 9096 9097 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 9098 if(name == "id") 9099 return allLowerCase ? name : "ID"; 9100 9101 char[160] buffer; 9102 int bufferIndex = 0; 9103 bool shouldCap = true; 9104 bool shouldSpace; 9105 bool lastWasCap; 9106 foreach(idx, char ch; name) { 9107 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9108 9109 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 9110 if(lastWasCap) { 9111 // two caps in a row, don't change. Prolly acronym. 9112 } else { 9113 if(idx) 9114 shouldSpace = true; // new word, add space 9115 } 9116 9117 lastWasCap = true; 9118 } else { 9119 lastWasCap = false; 9120 } 9121 9122 if(shouldSpace) { 9123 buffer[bufferIndex++] = space; 9124 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 9125 shouldSpace = false; 9126 } 9127 if(shouldCap) { 9128 if(ch >= 'a' && ch <= 'z') 9129 ch -= 32; 9130 shouldCap = false; 9131 } 9132 if(allLowerCase && ch >= 'A' && ch <= 'Z') 9133 ch += 32; 9134 buffer[bufferIndex++] = ch; 9135 } 9136 return buffer[0 .. bufferIndex].idup; 9137 } 9138 9139 /* 9140 string urlFor(alias func)() { 9141 return __traits(identifier, func); 9142 } 9143 */ 9144 9145 /++ 9146 UDA: The name displayed to the user in auto-generated HTML. 9147 9148 Default is `beautify(identifier)`. 9149 +/ 9150 struct DisplayName { 9151 string name; 9152 } 9153 9154 /++ 9155 UDA: The name used in the URL or web parameter. 9156 9157 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 9158 +/ 9159 struct UrlName { 9160 string name; 9161 } 9162 9163 /++ 9164 UDA: default format to respond for this method 9165 +/ 9166 struct DefaultFormat { string value; } 9167 9168 class MissingArgumentException : Exception { 9169 string functionName; 9170 string argumentName; 9171 string argumentType; 9172 9173 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9174 this.functionName = functionName; 9175 this.argumentName = argumentName; 9176 this.argumentType = argumentType; 9177 9178 super("Missing Argument: " ~ this.argumentName, file, line, next); 9179 } 9180 } 9181 9182 /++ 9183 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 9184 9185 History: 9186 Added December 15, 2021 (dub v10.5) 9187 +/ 9188 class ResourceNotFoundException : Exception { 9189 string resourceType; 9190 string resourceId; 9191 9192 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 9193 this.resourceType = resourceType; 9194 this.resourceId = resourceId; 9195 9196 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 9197 } 9198 9199 } 9200 9201 /++ 9202 This can be attached to any constructor or function called from the cgi system. 9203 9204 If it is present, the function argument can NOT be set from web params, but instead 9205 is set to the return value of the given `func`. 9206 9207 If `func` can take a parameter of type [Cgi], it will be passed the one representing 9208 the current request. Otherwise, it must take zero arguments. 9209 9210 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 9211 for the connection. Any of type [Session] (with an argument) is also assumed to come from 9212 the cgi object. 9213 9214 const arguments are also supported. 9215 +/ 9216 struct ifCalledFromWeb(alias func) {} 9217 9218 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 9219 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 9220 9221 // FIXME: any array of structs should also be settable or gettable from csv as well. 9222 9223 // FIXME: think more about checkboxes and bools. 9224 9225 import std.traits; 9226 9227 Parameters!method params; 9228 alias idents = ParameterIdentifierTuple!method; 9229 alias defaults = ParameterDefaults!method; 9230 9231 const(string)[] names; 9232 const(string)[] values; 9233 9234 // first, check for missing arguments and initialize to defaults if necessary 9235 9236 static if(is(typeof(method) P == __parameters)) 9237 foreach(idx, param; P) {{ 9238 // see: mustNotBeSetFromWebParams 9239 static if(is(param : Cgi)) { 9240 static assert(!is(param == immutable)); 9241 cast() params[idx] = cgi; 9242 } else static if(is(param == Session!D, D)) { 9243 static assert(!is(param == immutable)); 9244 cast() params[idx] = cgi.getSessionObject!D(); 9245 } else { 9246 bool populated; 9247 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 9248 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9249 static if(is(typeof(func(cgi)))) 9250 params[idx] = func(cgi); 9251 else 9252 params[idx] = func(); 9253 9254 populated = true; 9255 } 9256 } 9257 9258 if(!populated) { 9259 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 9260 params[idx] = param.getAutomaticallyForCgi(cgi); 9261 populated = true; 9262 } 9263 } 9264 9265 if(!populated) { 9266 auto ident = idents[idx]; 9267 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9268 if(ident !in cgi.get) { 9269 static if(is(defaults[idx] == void)) { 9270 static if(is(param == bool)) 9271 params[idx] = false; 9272 else 9273 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9274 } else 9275 params[idx] = defaults[idx]; 9276 } 9277 } else { 9278 if(ident !in cgi.post) { 9279 static if(is(defaults[idx] == void)) { 9280 static if(is(param == bool)) 9281 params[idx] = false; 9282 else 9283 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 9284 } else 9285 params[idx] = defaults[idx]; 9286 } 9287 } 9288 } 9289 } 9290 }} 9291 9292 // second, parse the arguments in order to build up arrays, etc. 9293 9294 static bool setVariable(T)(string name, string paramName, T* what, string value) { 9295 static if(is(T == struct)) { 9296 if(name == paramName) { 9297 *what = T.init; 9298 return true; 9299 } else { 9300 // could be a child. gonna allow either obj.field OR obj[field] 9301 9302 string afterName; 9303 9304 if(name[paramName.length] == '[') { 9305 int count = 1; 9306 auto idx = paramName.length + 1; 9307 while(idx < name.length && count > 0) { 9308 if(name[idx] == '[') 9309 count++; 9310 else if(name[idx] == ']') { 9311 count--; 9312 if(count == 0) break; 9313 } 9314 idx++; 9315 } 9316 9317 if(idx == name.length) 9318 return false; // malformed 9319 9320 auto insideBrackets = name[paramName.length + 1 .. idx]; 9321 afterName = name[idx + 1 .. $]; 9322 9323 name = name[0 .. paramName.length]; 9324 9325 paramName = insideBrackets; 9326 9327 } else if(name[paramName.length] == '.') { 9328 paramName = name[paramName.length + 1 .. $]; 9329 name = paramName; 9330 int p = 0; 9331 foreach(ch; paramName) { 9332 if(ch == '.' || ch == '[') 9333 break; 9334 p++; 9335 } 9336 9337 afterName = paramName[p .. $]; 9338 paramName = paramName[0 .. p]; 9339 } else { 9340 return false; 9341 } 9342 9343 if(paramName.length) 9344 // set the child member 9345 switch(paramName) { 9346 foreach(idx, memberName; __traits(allMembers, T)) 9347 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9348 // data member! 9349 case memberName: 9350 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 9351 } 9352 default: 9353 // ok, not a member 9354 } 9355 } 9356 9357 return false; 9358 } else static if(is(T == enum)) { 9359 *what = to!T(value); 9360 return true; 9361 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9362 *what = to!T(value); 9363 return true; 9364 } else static if(is(T == bool)) { 9365 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 9366 return true; 9367 } else static if(is(T == K[], K)) { 9368 K tmp; 9369 if(name == paramName) { 9370 // direct - set and append 9371 if(setVariable(name, paramName, &tmp, value)) { 9372 (*what) ~= tmp; 9373 return true; 9374 } else { 9375 return false; 9376 } 9377 } else { 9378 // child, append to last element 9379 // FIXME: what about range violations??? 9380 auto ptr = &(*what)[(*what).length - 1]; 9381 return setVariable(name, paramName, ptr, value); 9382 9383 } 9384 } else static if(is(T == V[K], K, V)) { 9385 // assoc array, name[key] is valid 9386 if(name == paramName) { 9387 // no action necessary 9388 return true; 9389 } else if(name[paramName.length] == '[') { 9390 int count = 1; 9391 auto idx = paramName.length + 1; 9392 while(idx < name.length && count > 0) { 9393 if(name[idx] == '[') 9394 count++; 9395 else if(name[idx] == ']') { 9396 count--; 9397 if(count == 0) break; 9398 } 9399 idx++; 9400 } 9401 if(idx == name.length) 9402 return false; // malformed 9403 9404 auto insideBrackets = name[paramName.length + 1 .. idx]; 9405 auto afterName = name[idx + 1 .. $]; 9406 9407 auto k = to!K(insideBrackets); 9408 V v; 9409 if(auto ptr = k in *what) 9410 v = *ptr; 9411 9412 name = name[0 .. paramName.length]; 9413 //writeln(name, afterName, " ", paramName); 9414 9415 auto ret = setVariable(name ~ afterName, paramName, &v, value); 9416 if(ret) { 9417 (*what)[k] = v; 9418 return true; 9419 } 9420 } 9421 9422 return false; 9423 } else { 9424 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9425 } 9426 9427 //return false; 9428 } 9429 9430 void setArgument(string name, string value) { 9431 int p; 9432 foreach(ch; name) { 9433 if(ch == '.' || ch == '[') 9434 break; 9435 p++; 9436 } 9437 9438 auto paramName = name[0 .. p]; 9439 9440 sw: switch(paramName) { 9441 static if(is(typeof(method) P == __parameters)) 9442 foreach(idx, param; P) { 9443 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 9444 // cannot be set from the outside 9445 } else { 9446 case idents[idx]: 9447 static if(is(param == Cgi.UploadedFile)) { 9448 params[idx] = cgi.files[name]; 9449 } else static if(is(param : const Cgi.UploadedFile[])) { 9450 (cast() params[idx]) = cgi.filesArray[name]; 9451 } else { 9452 setVariable(name, paramName, ¶ms[idx], value); 9453 } 9454 break sw; 9455 } 9456 } 9457 default: 9458 // ignore; not relevant argument 9459 } 9460 } 9461 9462 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 9463 names = cgi.allGetNamesInOrder; 9464 values = cgi.allGetValuesInOrder; 9465 } else { 9466 names = cgi.allPostNamesInOrder; 9467 values = cgi.allPostValuesInOrder; 9468 } 9469 9470 foreach(idx, name; names) { 9471 setArgument(name, values[idx]); 9472 } 9473 9474 static if(is(ReturnType!method == void)) { 9475 typeof(null) ret; 9476 dg(params); 9477 } else { 9478 auto ret = dg(params); 9479 } 9480 9481 // FIXME: format return values 9482 // options are: json, html, csv. 9483 // also may need to wrap in envelope format: none, html, or json. 9484 return ret; 9485 } 9486 9487 private bool mustNotBeSetFromWebParams(T, attrs...)() { 9488 static if(is(T : const(Cgi))) { 9489 return true; 9490 } else static if(is(T : const(Session!D), D)) { 9491 return true; 9492 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 9493 return true; 9494 } else { 9495 foreach(uda; attrs) 9496 static if(is(uda == ifCalledFromWeb!func, alias func)) 9497 return true; 9498 return false; 9499 } 9500 } 9501 9502 private bool hasIfCalledFromWeb(attrs...)() { 9503 foreach(uda; attrs) 9504 static if(is(uda == ifCalledFromWeb!func, alias func)) 9505 return true; 9506 return false; 9507 } 9508 9509 /++ 9510 Implies POST path for the thing itself, then GET will get the automatic form. 9511 9512 The given customizer, if present, will be called as a filter on the Form object. 9513 9514 History: 9515 Added December 27, 2020 9516 +/ 9517 template AutomaticForm(alias customizer) { } 9518 9519 /++ 9520 This is meant to be returned by a function that takes a form POST submission. You 9521 want to set the url of the new resource it created, which is set as the http 9522 Location header for a "201 Created" result, and you can also set a separate 9523 destination for browser users, which it sets via a "Refresh" header. 9524 9525 The `resourceRepresentation` should generally be the thing you just created, and 9526 it will be the body of the http response when formatted through the presenter. 9527 The exact thing is up to you - it could just return an id, or the whole object, or 9528 perhaps a partial object. 9529 9530 Examples: 9531 --- 9532 class Test : WebObject { 9533 @(Cgi.RequestMethod.POST) 9534 CreatedResource!int makeThing(string value) { 9535 return CreatedResource!int(value.to!int, "/resources/id"); 9536 } 9537 } 9538 --- 9539 9540 History: 9541 Added December 18, 2021 9542 +/ 9543 struct CreatedResource(T) { 9544 static if(!is(T == void)) 9545 T resourceRepresentation; 9546 string resourceUrl; 9547 string refreshUrl; 9548 } 9549 9550 /+ 9551 /++ 9552 This can be attached as a UDA to a handler to add a http Refresh header on a 9553 successful run. (It will not be attached if the function throws an exception.) 9554 This will refresh the browser the given number of seconds after the page loads, 9555 to the url returned by `urlFunc`, which can be either a static function or a 9556 member method of the current handler object. 9557 9558 You might use this for a POST handler that is normally used from ajax, but you 9559 want it to degrade gracefully to a temporarily flashed message before reloading 9560 the main page. 9561 9562 History: 9563 Added December 18, 2021 9564 +/ 9565 struct Refresh(alias urlFunc) { 9566 int waitInSeconds; 9567 9568 string url() { 9569 static if(__traits(isStaticFunction, urlFunc)) 9570 return urlFunc(); 9571 else static if(is(urlFunc : string)) 9572 return urlFunc; 9573 } 9574 } 9575 +/ 9576 9577 /+ 9578 /++ 9579 Sets a filter to be run before 9580 9581 A before function can do validations of params and log and stop the function from running. 9582 +/ 9583 template Before(alias b) {} 9584 template After(alias b) {} 9585 +/ 9586 9587 /+ 9588 Argument conversions: for the most part, it is to!Thing(string). 9589 9590 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9591 they are passed 9592 9593 arr=foo&arr=bar <-- notice the same name. 9594 9595 Structs are first declared with an empty thing, then have their members set individually, 9596 with dot notation. The members are not required, just the initial declaration. 9597 9598 struct Foo { 9599 int a; 9600 string b; 9601 } 9602 void test(Foo foo){} 9603 9604 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9605 9606 Arrays of structs use this declaration. 9607 9608 void test(Foo[] foo) {} 9609 9610 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9611 9612 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9613 declaration is ignored. 9614 9615 Mind that order matters! The declaration MUST come first in the string. 9616 9617 Arrays of struct members follow this rule recursively. 9618 9619 struct Foo { 9620 int[] a; 9621 } 9622 9623 foo&foo.a=1&foo.a=2&foo&foo.a=1 9624 9625 9626 Associative arrays are formatted with brackets, after a declaration, like structs: 9627 9628 foo&foo[key]=value&foo[other_key]=value 9629 9630 9631 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9632 do not support the strict ordering requirements to work with these struct protocols. 9633 9634 FIXME: also perhaps accept application/json to better work with outside trash. 9635 9636 9637 Return values are also auto-formatted according to user-requested type: 9638 for json, it loops over and converts. 9639 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9640 +/ 9641 9642 /++ 9643 A web presenter is responsible for rendering things to HTML to be usable 9644 in a web browser. 9645 9646 They are passed as template arguments to the base classes of [WebObject] 9647 9648 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9649 and override it. Use forwarding and specialization to customize it. 9650 9651 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9652 class templates and your overridden templates work with each other. 9653 9654 --- 9655 class MyPresenter : WebPresenter!(MyPresenter) { 9656 @Override 9657 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9658 // present the CustomType 9659 } 9660 @Override 9661 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9662 // handle everything else via the super class, which will call 9663 // back to your class when appropriate 9664 super.presentSuccessfulReturnAsHtml(cgi, ret); 9665 } 9666 } 9667 --- 9668 9669 The meta argument in there can be overridden by your own facility. 9670 9671 +/ 9672 class WebPresenter(CRTP) { 9673 9674 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9675 /// If you override a plain method, use `override`. If a template, use `@Override`. 9676 enum Override; 9677 9678 string script() { 9679 return ` 9680 `; 9681 } 9682 9683 string style() { 9684 return ` 9685 :root { 9686 --mild-border: #ccc; 9687 --middle-border: #999; 9688 --accent-color: #f2f2f2; 9689 --sidebar-color: #fefefe; 9690 } 9691 ` ~ genericFormStyling() ~ genericSiteStyling(); 9692 } 9693 9694 string genericFormStyling() { 9695 return 9696 q"css 9697 table.automatic-data-display { 9698 border-collapse: collapse; 9699 border: solid 1px var(--mild-border); 9700 } 9701 9702 table.automatic-data-display td { 9703 vertical-align: top; 9704 border: solid 1px var(--mild-border); 9705 padding: 2px 4px; 9706 } 9707 9708 table.automatic-data-display th { 9709 border: solid 1px var(--mild-border); 9710 border-bottom: solid 1px var(--middle-border); 9711 padding: 2px 4px; 9712 } 9713 9714 ol.automatic-data-display { 9715 margin: 0px; 9716 list-style-position: inside; 9717 padding: 0px; 9718 } 9719 9720 dl.automatic-data-display { 9721 9722 } 9723 9724 .automatic-form { 9725 max-width: 600px; 9726 } 9727 9728 .form-field { 9729 margin: 0.5em; 9730 padding-left: 0.5em; 9731 } 9732 9733 .label-text { 9734 display: block; 9735 font-weight: bold; 9736 margin-left: -0.5em; 9737 } 9738 9739 .submit-button-holder { 9740 padding-left: 2em; 9741 } 9742 9743 .add-array-button { 9744 9745 } 9746 css"; 9747 } 9748 9749 string genericSiteStyling() { 9750 return 9751 q"css 9752 * { box-sizing: border-box; } 9753 html, body { margin: 0px; } 9754 body { 9755 font-family: sans-serif; 9756 } 9757 header { 9758 background: var(--accent-color); 9759 height: 64px; 9760 } 9761 footer { 9762 background: var(--accent-color); 9763 height: 64px; 9764 } 9765 #site-container { 9766 display: flex; 9767 } 9768 main { 9769 flex: 1 1 auto; 9770 order: 2; 9771 min-height: calc(100vh - 64px - 64px); 9772 padding: 4px; 9773 padding-left: 1em; 9774 } 9775 #sidebar { 9776 flex: 0 0 16em; 9777 order: 1; 9778 background: var(--sidebar-color); 9779 } 9780 css"; 9781 } 9782 9783 import arsd.dom; 9784 Element htmlContainer() { 9785 auto document = new Document(q"html 9786 <!DOCTYPE html> 9787 <html class="no-script"> 9788 <head> 9789 <script>document.documentElement.classList.remove("no-script");</script> 9790 <style>.no-script requires-script { display: none; }</style> 9791 <title>D Application</title> 9792 <meta name="viewport" content="initial-scale=1, width=device-width" /> 9793 <link rel="stylesheet" href="style.css" /> 9794 </head> 9795 <body> 9796 <header></header> 9797 <div id="site-container"> 9798 <main></main> 9799 <div id="sidebar"></div> 9800 </div> 9801 <footer></footer> 9802 <script src="script.js"></script> 9803 </body> 9804 </html> 9805 html", true, true); 9806 9807 return document.requireSelector("main"); 9808 } 9809 9810 /// Renders a response as an HTTP error with associated html body 9811 void renderBasicError(Cgi cgi, int httpErrorCode) { 9812 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9813 auto c = htmlContainer(); 9814 c.innerText = getHttpCodeText(httpErrorCode); 9815 cgi.setResponseContentType("text/html; charset=utf-8"); 9816 cgi.write(c.parentDocument.toString(), true); 9817 } 9818 9819 template methodMeta(alias method) { 9820 enum methodMeta = null; 9821 } 9822 9823 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9824 switch(format) { 9825 case "html": 9826 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9827 break; 9828 case "json": 9829 import arsd.jsvar; 9830 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 9831 var json; 9832 foreach(index, type; Types) { 9833 if(ret.contains == index) 9834 json = ret.payload[index]; 9835 } 9836 } else { 9837 var json = ret; 9838 } 9839 var envelope = json; // var.emptyObject; 9840 /* 9841 envelope.success = true; 9842 envelope.result = json; 9843 envelope.error = null; 9844 */ 9845 cgi.setResponseContentType("application/json"); 9846 cgi.write(envelope.toJson(), true); 9847 break; 9848 default: 9849 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 9850 } 9851 } 9852 9853 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9854 /// in the default presenter - allowing the function to have full low-level control over the 9855 /// response. 9856 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9857 // nothing intentionally! 9858 } 9859 9860 /// Redirections are forwarded to [Cgi.setResponseLocation] 9861 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9862 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9863 } 9864 9865 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9866 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9867 cgi.setResponseStatus(getHttpCodeText(201)); 9868 if(ret.resourceUrl.length) 9869 cgi.header("Location: " ~ ret.resourceUrl); 9870 if(ret.refreshUrl.length) 9871 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9872 static if(!is(R == void)) 9873 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9874 } 9875 9876 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9877 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9878 bool outputted = false; 9879 foreach(index, type; Types) { 9880 if(ret.contains == index) { 9881 assert(!outputted); 9882 outputted = true; 9883 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9884 } 9885 } 9886 if(!outputted) 9887 assert(0); 9888 } 9889 9890 /++ 9891 An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort if the filename member is non-null of the FileResource interface. 9892 +/ 9893 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9894 cgi.setCache(true); // not necessarily true but meh 9895 if(auto fn = ret.filename()) { 9896 cgi.header("Content-Disposition: attachment; filename="~fn~";"); 9897 } 9898 cgi.setResponseContentType(ret.contentType); 9899 cgi.write(ret.getData(), true); 9900 } 9901 9902 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9903 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9904 auto container = this.htmlContainer(); 9905 container.appendChild(formatReturnValueAsHtml(ret)); 9906 cgi.write(container.parentDocument.toString(), true); 9907 } 9908 9909 /++ 9910 9911 History: 9912 Added January 23, 2023 (dub v11.0) 9913 +/ 9914 void presentExceptionalReturn(Meta)(Cgi cgi, Throwable t, Meta meta, string format) { 9915 switch(format) { 9916 case "html": 9917 presentExceptionAsHtml(cgi, t, meta); 9918 break; 9919 case "json": 9920 presentExceptionAsJsonImpl(cgi, t); 9921 break; 9922 default: 9923 } 9924 } 9925 9926 private void presentExceptionAsJsonImpl()(Cgi cgi, Throwable t) { 9927 cgi.setResponseStatus("500 Internal Server Error"); 9928 cgi.setResponseContentType("application/json"); 9929 import arsd.jsvar; 9930 var v = var.emptyObject; 9931 v.type = typeid(t).toString; 9932 v.msg = t.msg; 9933 v.fullString = t.toString(); 9934 cgi.write(v.toJson(), true); 9935 } 9936 9937 9938 /++ 9939 If you override this, you will need to cast the exception type `t` dynamically, 9940 but can then use the template arguments here to refer back to the function. 9941 9942 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 9943 method on the live object. You could, in theory, change arguments and retry, but I 9944 provide that information mostly with the expectation that you will use them to make 9945 useful forms or richer error messages for the user. 9946 9947 History: 9948 BREAKING CHANGE on January 23, 2023 (v11.0 ): it previously took an `alias func` and `T dg` to call the function again. 9949 I removed this in favor of a `Meta` param. 9950 9951 Before: `void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg)` 9952 9953 After: `void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta)` 9954 9955 If you used the func for something, move that something into your `methodMeta` template. 9956 9957 What is the benefit of this change? Somewhat smaller executables and faster builds thanks to more reused functions, together with 9958 enabling an easier implementation of [presentExceptionalReturn]. 9959 +/ 9960 void presentExceptionAsHtml(Meta)(Cgi cgi, Throwable t, Meta meta) { 9961 Form af; 9962 /+ 9963 foreach(attr; __traits(getAttributes, func)) { 9964 static if(__traits(isSame, attr, AutomaticForm)) { 9965 af = createAutomaticFormForFunction!(func)(dg); 9966 } 9967 } 9968 +/ 9969 presentExceptionAsHtmlImpl(cgi, t, af); 9970 } 9971 9972 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 9973 if(auto e = cast(ResourceNotFoundException) t) { 9974 auto container = this.htmlContainer(); 9975 9976 container.addChild("p", e.msg); 9977 9978 if(!cgi.outputtedResponseData) 9979 cgi.setResponseStatus("404 Not Found"); 9980 cgi.write(container.parentDocument.toString(), true); 9981 } else if(auto mae = cast(MissingArgumentException) t) { 9982 if(automaticForm is null) 9983 goto generic; 9984 auto container = this.htmlContainer(); 9985 if(cgi.requestMethod == Cgi.RequestMethod.POST) 9986 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 9987 container.appendChild(automaticForm); 9988 9989 cgi.write(container.parentDocument.toString(), true); 9990 } else { 9991 generic: 9992 auto container = this.htmlContainer(); 9993 9994 // import std.stdio; writeln(t.toString()); 9995 9996 container.appendChild(exceptionToElement(t)); 9997 9998 container.addChild("h4", "GET"); 9999 foreach(k, v; cgi.get) { 10000 auto deets = container.addChild("details"); 10001 deets.addChild("summary", k); 10002 deets.addChild("div", v); 10003 } 10004 10005 container.addChild("h4", "POST"); 10006 foreach(k, v; cgi.post) { 10007 auto deets = container.addChild("details"); 10008 deets.addChild("summary", k); 10009 deets.addChild("div", v); 10010 } 10011 10012 10013 if(!cgi.outputtedResponseData) 10014 cgi.setResponseStatus("500 Internal Server Error"); 10015 cgi.write(container.parentDocument.toString(), true); 10016 } 10017 } 10018 10019 Element exceptionToElement(Throwable t) { 10020 auto div = Element.make("div"); 10021 div.addClass("exception-display"); 10022 10023 div.addChild("p", t.msg); 10024 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 10025 10026 auto pre = div.addChild("pre"); 10027 string s; 10028 s = t.toString(); 10029 Element currentBox; 10030 bool on = false; 10031 foreach(line; s.splitLines) { 10032 if(!on && line.startsWith("-----")) 10033 on = true; 10034 if(!on) continue; 10035 if(line.indexOf("arsd/") != -1) { 10036 if(currentBox is null) { 10037 currentBox = pre.addChild("details"); 10038 currentBox.addChild("summary", "Framework code"); 10039 } 10040 currentBox.addChild("span", line ~ "\n"); 10041 } else { 10042 pre.addChild("span", line ~ "\n"); 10043 currentBox = null; 10044 } 10045 } 10046 10047 return div; 10048 } 10049 10050 /++ 10051 Returns an element for a particular type 10052 +/ 10053 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 10054 import std.traits; 10055 10056 auto div = Element.make("div"); 10057 div.addClass("form-field"); 10058 10059 static if(is(T : const Cgi.UploadedFile)) { 10060 Element lbl; 10061 if(displayName !is null) { 10062 lbl = div.addChild("label"); 10063 lbl.addChild("span", displayName, "label-text"); 10064 lbl.appendText(" "); 10065 } else { 10066 lbl = div; 10067 } 10068 auto i = lbl.addChild("input", name); 10069 i.attrs.name = name; 10070 i.attrs.type = "file"; 10071 i.attrs.multiple = "multiple"; 10072 } else static if(is(T == Cgi.UploadedFile)) { 10073 Element lbl; 10074 if(displayName !is null) { 10075 lbl = div.addChild("label"); 10076 lbl.addChild("span", displayName, "label-text"); 10077 lbl.appendText(" "); 10078 } else { 10079 lbl = div; 10080 } 10081 auto i = lbl.addChild("input", name); 10082 i.attrs.name = name; 10083 i.attrs.type = "file"; 10084 } else static if(is(T == enum)) { 10085 Element lbl; 10086 if(displayName !is null) { 10087 lbl = div.addChild("label"); 10088 lbl.addChild("span", displayName, "label-text"); 10089 lbl.appendText(" "); 10090 } else { 10091 lbl = div; 10092 } 10093 auto i = lbl.addChild("select", name); 10094 i.attrs.name = name; 10095 10096 foreach(memberName; __traits(allMembers, T)) 10097 i.addChild("option", memberName); 10098 10099 } else static if(is(T == struct)) { 10100 if(displayName !is null) 10101 div.addChild("span", displayName, "label-text"); 10102 auto fieldset = div.addChild("fieldset"); 10103 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 10104 fieldset.addChild("input", name); 10105 foreach(idx, memberName; __traits(allMembers, T)) 10106 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10107 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 10108 } 10109 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 10110 Element lbl; 10111 if(displayName !is null) { 10112 lbl = div.addChild("label"); 10113 lbl.addChild("span", displayName, "label-text"); 10114 lbl.appendText(" "); 10115 } else { 10116 lbl = div; 10117 } 10118 Element i; 10119 if(udaSuggestion) { 10120 i = udaSuggestion(); 10121 lbl.appendChild(i); 10122 } else { 10123 i = lbl.addChild("input", name); 10124 } 10125 i.attrs.name = name; 10126 static if(isSomeString!T) 10127 i.attrs.type = "text"; 10128 else 10129 i.attrs.type = "number"; 10130 if(i.tagName == "textarea") 10131 i.textContent = to!string(T.init); 10132 else 10133 i.attrs.value = to!string(T.init); 10134 } else static if(is(T == bool)) { 10135 Element lbl; 10136 if(displayName !is null) { 10137 lbl = div.addChild("label"); 10138 lbl.addChild("span", displayName, "label-text"); 10139 lbl.appendText(" "); 10140 } else { 10141 lbl = div; 10142 } 10143 auto i = lbl.addChild("input", name); 10144 i.attrs.type = "checkbox"; 10145 i.attrs.value = "true"; 10146 i.attrs.name = name; 10147 } else static if(is(T == K[], K)) { 10148 auto templ = div.addChild("template"); 10149 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 10150 if(displayName !is null) 10151 div.addChild("span", displayName, "label-text"); 10152 auto btn = div.addChild("button"); 10153 btn.addClass("add-array-button"); 10154 btn.attrs.type = "button"; 10155 btn.innerText = "Add"; 10156 btn.attrs.onclick = q{ 10157 var a = document.importNode(this.parentNode.firstChild.content, true); 10158 this.parentNode.insertBefore(a, this); 10159 }; 10160 } else static if(is(T == V[K], K, V)) { 10161 div.innerText = "assoc array not implemented for automatic form at this time"; 10162 } else { 10163 static assert(0, "unsupported type for cgi call " ~ T.stringof); 10164 } 10165 10166 10167 return div; 10168 } 10169 10170 /// creates a form for gathering the function's arguments 10171 Form createAutomaticFormForFunction(alias method, T)(T dg) { 10172 10173 auto form = cast(Form) Element.make("form"); 10174 10175 form.method = "POST"; // FIXME 10176 10177 form.addClass("automatic-form"); 10178 10179 string formDisplayName = beautify(__traits(identifier, method)); 10180 foreach(attr; __traits(getAttributes, method)) 10181 static if(is(typeof(attr) == DisplayName)) 10182 formDisplayName = attr.name; 10183 form.addChild("h3", formDisplayName); 10184 10185 import std.traits; 10186 10187 //Parameters!method params; 10188 //alias idents = ParameterIdentifierTuple!method; 10189 //alias defaults = ParameterDefaults!method; 10190 10191 static if(is(typeof(method) P == __parameters)) 10192 foreach(idx, _; P) {{ 10193 10194 alias param = P[idx .. idx + 1]; 10195 10196 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 10197 string displayName = beautify(__traits(identifier, param)); 10198 Element function() element; 10199 foreach(attr; __traits(getAttributes, param)) { 10200 static if(is(typeof(attr) == DisplayName)) 10201 displayName = attr.name; 10202 else static if(is(typeof(attr) : typeof(element))) { 10203 element = attr; 10204 } 10205 } 10206 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 10207 if(i.querySelector("input[type=file]") !is null) 10208 form.setAttribute("enctype", "multipart/form-data"); 10209 } 10210 }} 10211 10212 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10213 10214 return form; 10215 } 10216 10217 /// creates a form for gathering object members (for the REST object thing right now) 10218 Form createAutomaticFormForObject(T)(T obj) { 10219 auto form = cast(Form) Element.make("form"); 10220 10221 form.addClass("automatic-form"); 10222 10223 form.addChild("h3", beautify(__traits(identifier, T))); 10224 10225 import std.traits; 10226 10227 //Parameters!method params; 10228 //alias idents = ParameterIdentifierTuple!method; 10229 //alias defaults = ParameterDefaults!method; 10230 10231 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 10232 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10233 string displayName = beautify(memberName); 10234 Element function() element; 10235 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 10236 static if(is(typeof(attr) == DisplayName)) 10237 displayName = attr.name; 10238 else static if(is(typeof(attr) : typeof(element))) 10239 element = attr; 10240 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 10241 10242 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 10243 }}} 10244 10245 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 10246 10247 return form; 10248 } 10249 10250 /// 10251 Element formatReturnValueAsHtml(T)(T t) { 10252 import std.traits; 10253 10254 static if(is(T == typeof(null))) { 10255 return Element.make("span"); 10256 } else static if(is(T : Element)) { 10257 return t; 10258 } else static if(is(T == MultipleResponses!Types, Types...)) { 10259 foreach(index, type; Types) { 10260 if(t.contains == index) 10261 return formatReturnValueAsHtml(t.payload[index]); 10262 } 10263 assert(0); 10264 } else static if(is(T == Paginated!E, E)) { 10265 auto e = Element.make("div").addClass("paginated-result"); 10266 e.appendChild(formatReturnValueAsHtml(t.items)); 10267 if(t.nextPageUrl.length) 10268 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 10269 return e; 10270 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 10271 return Element.make("span", to!string(t), "automatic-data-display"); 10272 } else static if(is(T == V[K], K, V)) { 10273 auto dl = Element.make("dl"); 10274 dl.addClass("automatic-data-display associative-array"); 10275 foreach(k, v; t) { 10276 dl.addChild("dt", to!string(k)); 10277 dl.addChild("dd", formatReturnValueAsHtml(v)); 10278 } 10279 return dl; 10280 } else static if(is(T == struct)) { 10281 auto dl = Element.make("dl"); 10282 dl.addClass("automatic-data-display struct"); 10283 10284 foreach(idx, memberName; __traits(allMembers, T)) 10285 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 10286 dl.addChild("dt", beautify(memberName)); 10287 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 10288 } 10289 10290 return dl; 10291 } else static if(is(T == bool)) { 10292 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 10293 } else static if(is(T == E[], E)) { 10294 static if(is(E : RestObject!Proxy, Proxy)) { 10295 // treat RestObject similar to struct 10296 auto table = cast(Table) Element.make("table"); 10297 table.addClass("automatic-data-display"); 10298 string[] names; 10299 foreach(idx, memberName; __traits(derivedMembers, E)) 10300 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10301 names ~= beautify(memberName); 10302 } 10303 table.appendHeaderRow(names); 10304 10305 foreach(l; t) { 10306 auto tr = table.appendRow(); 10307 foreach(idx, memberName; __traits(derivedMembers, E)) 10308 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10309 static if(memberName == "id") { 10310 string val = to!string(__traits(getMember, l, memberName)); 10311 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 10312 } else { 10313 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10314 } 10315 } 10316 } 10317 10318 return table; 10319 } else static if(is(E == struct)) { 10320 // an array of structs is kinda special in that I like 10321 // having those formatted as tables. 10322 auto table = cast(Table) Element.make("table"); 10323 table.addClass("automatic-data-display"); 10324 string[] names; 10325 foreach(idx, memberName; __traits(allMembers, E)) 10326 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10327 names ~= beautify(memberName); 10328 } 10329 table.appendHeaderRow(names); 10330 10331 foreach(l; t) { 10332 auto tr = table.appendRow(); 10333 foreach(idx, memberName; __traits(allMembers, E)) 10334 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 10335 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 10336 } 10337 } 10338 10339 return table; 10340 } else { 10341 // otherwise, I will just make a list. 10342 auto ol = Element.make("ol"); 10343 ol.addClass("automatic-data-display"); 10344 foreach(e; t) 10345 ol.addChild("li", formatReturnValueAsHtml(e)); 10346 return ol; 10347 } 10348 } else static if(is(T : Object)) { 10349 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 10350 return Element.make("div", t.toHtml()); 10351 else 10352 return Element.make("div", t.toString()); 10353 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 10354 10355 assert(0); 10356 } 10357 10358 } 10359 10360 /++ 10361 The base class for the [dispatcher] function and object support. 10362 +/ 10363 class WebObject { 10364 //protected Cgi cgi; 10365 10366 protected void initialize(Cgi cgi) { 10367 //this.cgi = cgi; 10368 } 10369 } 10370 10371 /++ 10372 Can return one of the given types, decided at runtime. The syntax 10373 is to declare all the possible types in the return value, then you 10374 can `return typeof(return)(...value...)` to construct it. 10375 10376 It has an auto-generated constructor for each value it can hold. 10377 10378 --- 10379 MultipleResponses!(Redirection, string) getData(int how) { 10380 if(how & 1) 10381 return typeof(return)(Redirection("http://dpldocs.info/")); 10382 else 10383 return typeof(return)("hi there!"); 10384 } 10385 --- 10386 10387 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 10388 +/ 10389 struct MultipleResponses(T...) { 10390 private size_t contains; 10391 private union { 10392 private T payload; 10393 } 10394 10395 static foreach(index, type; T) 10396 public this(type t) { 10397 contains = index; 10398 payload[index] = t; 10399 } 10400 10401 /++ 10402 This is primarily for testing. It is your way of getting to the response. 10403 10404 Let's say you wanted to test that one holding a Redirection and a string actually 10405 holds a string, by name of "test": 10406 10407 --- 10408 auto valueToTest = your_test_function(); 10409 10410 valueToTest.visit( 10411 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 10412 (string s) { assert(s == "test"); } // right value, go ahead and test it. 10413 ); 10414 --- 10415 10416 History: 10417 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 10418 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 10419 +/ 10420 void visit(Handlers...)(Handlers handlers) { 10421 template findHandler(type, int count, HandlersToCheck...) { 10422 static if(HandlersToCheck.length == 0) 10423 enum findHandler = -1; 10424 else { 10425 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 10426 enum findHandler = count; 10427 else 10428 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 10429 } 10430 } 10431 foreach(index, type; T) { 10432 enum handlerIndex = findHandler!(type, 0, Handlers); 10433 static if(handlerIndex == -1) 10434 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 10435 else { 10436 if(index == this.contains) 10437 handlers[handlerIndex](this.payload[index]); 10438 } 10439 } 10440 } 10441 10442 /+ 10443 auto toArsdJsvar()() { 10444 import arsd.jsvar; 10445 return var(null); 10446 } 10447 +/ 10448 } 10449 10450 // FIXME: implement this somewhere maybe 10451 struct RawResponse { 10452 int code; 10453 string[] headers; 10454 const(ubyte)[] responseBody; 10455 } 10456 10457 /++ 10458 You can return this from [WebObject] subclasses for redirections. 10459 10460 (though note the static types means that class must ALWAYS redirect if 10461 you return this directly. You might want to return [MultipleResponses] if it 10462 can be conditional) 10463 +/ 10464 struct Redirection { 10465 string to; /// The URL to redirect to. 10466 int code = 303; /// The HTTP code to return. 10467 } 10468 10469 /++ 10470 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 10471 10472 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 10473 the presenter in the dispatcher. 10474 10475 FIXME: explain this better 10476 10477 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 10478 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 10479 the runtime result of that is undefined. 10480 10481 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 10482 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 10483 10484 $(WARNING 10485 --- 10486 // legal in D, undefined runtime behavior with cgi.d, it may call either method 10487 // even if you put different URL udas on it, the current code ignores them. 10488 void foo(int a) {} 10489 void foo(string a) {} 10490 --- 10491 ) 10492 10493 See_Also: [serveRestObject], [serveStaticFile] 10494 +/ 10495 auto serveApi(T)(string urlPrefix) { 10496 assert(urlPrefix[$ - 1] == '/'); 10497 return serveApiInternal!T(urlPrefix); 10498 } 10499 10500 private string nextPieceFromSlash(ref string remainingUrl) { 10501 if(remainingUrl.length == 0) 10502 return remainingUrl; 10503 int slash = 0; 10504 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 10505 slash++; 10506 10507 // I am specifically passing `null` to differentiate it vs empty string 10508 // so in your ctor, `items` means new T(null) and `items/` means new T("") 10509 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 10510 // so if it is the last item, the dot can be used to load an alternative view 10511 // otherwise tho the dot is considered part of the identifier 10512 // FIXME 10513 10514 // again notice "" vs null here! 10515 if(slash == remainingUrl.length) 10516 remainingUrl = null; 10517 else 10518 remainingUrl = remainingUrl[slash + 1 .. $]; 10519 10520 return ident; 10521 } 10522 10523 /++ 10524 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 10525 +/ 10526 enum AddTrailingSlash; 10527 /// ditto 10528 enum RemoveTrailingSlash; 10529 10530 private auto serveApiInternal(T)(string urlPrefix) { 10531 10532 import arsd.dom; 10533 import arsd.jsvar; 10534 10535 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10536 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 10537 10538 try { 10539 // see duplicated code below by searching subresource_ctor 10540 // also see mustNotBeSetFromWebParams 10541 10542 static if(is(typeof(T.__ctor) P == __parameters)) { 10543 P params; 10544 10545 foreach(pidx, param; P) { 10546 static if(is(param : Cgi)) { 10547 static assert(!is(param == immutable)); 10548 cast() params[pidx] = cgi; 10549 } else static if(is(param == Session!D, D)) { 10550 static assert(!is(param == immutable)); 10551 cast() params[pidx] = cgi.getSessionObject!D(); 10552 10553 } else { 10554 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10555 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10556 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10557 static if(is(typeof(func(cgi)))) 10558 params[pidx] = func(cgi); 10559 else 10560 params[pidx] = func(); 10561 } 10562 } 10563 } else { 10564 10565 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10566 params[pidx] = param.getAutomaticallyForCgi(cgi); 10567 } else static if(is(param == string)) { 10568 auto ident = nextPieceFromSlash(remainingUrl); 10569 params[pidx] = ident; 10570 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10571 } 10572 } 10573 } 10574 10575 auto obj = new T(params); 10576 } else { 10577 auto obj = new T(); 10578 } 10579 10580 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 10581 } catch(Throwable t) { 10582 switch(cgi.request("format", "html")) { 10583 case "html": 10584 static void dummy() {} 10585 presenter.presentExceptionAsHtml(cgi, t, null); 10586 return true; 10587 case "json": 10588 var envelope = var.emptyObject; 10589 envelope.success = false; 10590 envelope.result = null; 10591 envelope.error = t.toString(); 10592 cgi.setResponseContentType("application/json"); 10593 cgi.write(envelope.toJson(), true); 10594 return true; 10595 default: 10596 throw t; 10597 // return true; 10598 } 10599 // return true; 10600 } 10601 10602 assert(0); 10603 } 10604 10605 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10606 10607 obj.initialize(cgi); 10608 10609 /+ 10610 Overload rules: 10611 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10612 statically. 10613 10614 Moreover, some args vs no args can be overloaded dynamically. 10615 +/ 10616 10617 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10618 /+ 10619 auto orig = remainingUrl; 10620 assert(0, 10621 (orig is null ? "__null" : orig) 10622 ~ " .. " ~ 10623 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10624 +/ 10625 10626 if(methodNameFromUrl is null) 10627 methodNameFromUrl = "__null"; 10628 10629 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10630 10631 if(remainingUrl.length) 10632 hack ~= "/"; 10633 10634 switch(hack) { 10635 foreach(methodName; __traits(derivedMembers, T)) 10636 static if(methodName != "__ctor") 10637 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10638 static if(is(typeof(overload) P == __parameters)) 10639 static if(is(typeof(overload) R == return)) 10640 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10641 { 10642 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10643 case urlNameForMethod: 10644 10645 static if(is(R : WebObject)) { 10646 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10647 10648 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10649 10650 // subresource_ctor 10651 // also see mustNotBeSetFromWebParams 10652 10653 P params; 10654 10655 string ident; 10656 10657 foreach(pidx, param; P) { 10658 static if(is(param : Cgi)) { 10659 static assert(!is(param == immutable)); 10660 cast() params[pidx] = cgi; 10661 } else static if(is(param == typeof(presenter))) { 10662 cast() param[pidx] = presenter; 10663 } else static if(is(param == Session!D, D)) { 10664 static assert(!is(param == immutable)); 10665 cast() params[pidx] = cgi.getSessionObject!D(); 10666 } else { 10667 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10668 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10669 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10670 static if(is(typeof(func(cgi)))) 10671 params[pidx] = func(cgi); 10672 else 10673 params[pidx] = func(); 10674 } 10675 } 10676 } else { 10677 10678 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10679 params[pidx] = param.getAutomaticallyForCgi(cgi); 10680 } else static if(is(param == string)) { 10681 ident = nextPieceFromSlash(remainingUrl); 10682 if(ident is null) { 10683 // trailing slash mandated on subresources 10684 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10685 return true; 10686 } else { 10687 params[pidx] = ident; 10688 } 10689 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10690 } 10691 } 10692 } 10693 10694 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10695 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10696 } else { 10697 // 404 it if any url left - not a subresource means we don't get to play with that! 10698 if(remainingUrl.length) 10699 return false; 10700 10701 bool automaticForm; 10702 10703 foreach(attr; __traits(getAttributes, overload)) 10704 static if(is(attr == AddTrailingSlash)) { 10705 if(remainingUrl is null) { 10706 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10707 return true; 10708 } 10709 } else static if(is(attr == RemoveTrailingSlash)) { 10710 if(remainingUrl !is null) { 10711 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10712 return true; 10713 } 10714 10715 } else static if(__traits(isSame, AutomaticForm, attr)) { 10716 automaticForm = true; 10717 } 10718 10719 /+ 10720 int zeroArgOverload = -1; 10721 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10722 bool calledWithZeroArgs = true; 10723 foreach(k, v; cgi.get) 10724 if(k != "format") { 10725 calledWithZeroArgs = false; 10726 break; 10727 } 10728 foreach(k, v; cgi.post) 10729 if(k != "format") { 10730 calledWithZeroArgs = false; 10731 break; 10732 } 10733 10734 // first, we need to go through and see if there is an empty one, since that 10735 // changes inside. But otherwise, all the stuff I care about can be done via 10736 // simple looping (other improper overloads might be flagged for runtime semantic check) 10737 // 10738 // an argument of type Cgi is ignored for these purposes 10739 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10740 static if(is(typeof(overload) P == __parameters)) 10741 static if(P.length == 0) 10742 zeroArgOverload = cast(int) idx; 10743 else static if(P.length == 1 && is(P[0] : Cgi)) 10744 zeroArgOverload = cast(int) idx; 10745 }} 10746 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10747 bool overloadHasBeenCalled = false; 10748 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10749 bool callFunction = true; 10750 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10751 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10752 callFunction = false; 10753 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10754 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10755 callFunction = false; 10756 10757 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10758 10759 bool hadAnyMethodRestrictions = false; 10760 bool foundAcceptableMethod = false; 10761 foreach(attr; __traits(getAttributes, overload)) { 10762 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10763 hadAnyMethodRestrictions = true; 10764 if(attr == cgi.requestMethod) 10765 foundAcceptableMethod = true; 10766 } 10767 } 10768 10769 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10770 callFunction = false; 10771 10772 /+ 10773 The overloads we really want to allow are the sane ones 10774 from the web perspective. Which is likely on HTTP verbs, 10775 for the most part, but might also be potentially based on 10776 some args vs zero args, or on argument names. Can't really 10777 do argument types very reliable through the web though; those 10778 should probably be different URLs. 10779 10780 Even names I feel is better done inside the function, so I'm not 10781 going to support that here. But the HTTP verbs and zero vs some 10782 args makes sense - it lets you define custom forms pretty easily. 10783 10784 Moreover, I'm of the opinion that empty overload really only makes 10785 sense on GET for this case. On a POST, it is just a missing argument 10786 exception and that should be handled by the presenter. But meh, I'll 10787 let the user define that, D only allows one empty arg thing anyway 10788 so the method UDAs are irrelevant. 10789 +/ 10790 if(callFunction) 10791 +/ 10792 10793 auto format = cgi.request("format", defaultFormat!overload()); 10794 auto wantsFormFormat = format.startsWith("form-"); 10795 10796 if(wantsFormFormat || (automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET)) { 10797 // Should I still show the form on a json thing? idk... 10798 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10799 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), wantsFormFormat ? format["form_".length .. $] : "html"); 10800 return true; 10801 } 10802 10803 try { 10804 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10805 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10806 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10807 } catch(Throwable t) { 10808 // presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10809 presenter.presentExceptionalReturn(cgi, t, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), format); 10810 } 10811 return true; 10812 //}} 10813 10814 //cgi.header("Accept: POST"); // FIXME list the real thing 10815 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10816 //return true; 10817 } 10818 } 10819 } 10820 case "GET script.js": 10821 cgi.setResponseContentType("text/javascript"); 10822 cgi.gzipResponse = true; 10823 cgi.write(presenter.script(), true); 10824 return true; 10825 case "GET style.css": 10826 cgi.setResponseContentType("text/css"); 10827 cgi.gzipResponse = true; 10828 cgi.write(presenter.style(), true); 10829 return true; 10830 default: 10831 return false; 10832 } 10833 10834 assert(0); 10835 } 10836 return DispatcherDefinition!internalHandler(urlPrefix, false); 10837 } 10838 10839 string defaultFormat(alias method)() { 10840 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10841 foreach(attr; __traits(getAttributes, method)) { 10842 static if(is(typeof(attr) == DefaultFormat)) { 10843 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10844 return attr.value; 10845 } 10846 } 10847 return "html"; 10848 } 10849 10850 struct Paginated(T) { 10851 T[] items; 10852 string nextPageUrl; 10853 } 10854 10855 template urlNamesForMethod(alias method, string default_) { 10856 string[] helper() { 10857 auto verb = Cgi.RequestMethod.GET; 10858 bool foundVerb = false; 10859 bool foundNoun = false; 10860 10861 string def = default_; 10862 10863 bool hasAutomaticForm = false; 10864 10865 foreach(attr; __traits(getAttributes, method)) { 10866 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10867 verb = attr; 10868 if(foundVerb) 10869 assert(0, "Multiple http verbs on one function is not currently supported"); 10870 foundVerb = true; 10871 } 10872 static if(is(typeof(attr) == UrlName)) { 10873 if(foundNoun) 10874 assert(0, "Multiple url names on one function is not currently supported"); 10875 foundNoun = true; 10876 def = attr.name; 10877 } 10878 static if(__traits(isSame, attr, AutomaticForm)) { 10879 hasAutomaticForm = true; 10880 } 10881 } 10882 10883 if(def is null) 10884 def = "__null"; 10885 10886 string[] ret; 10887 10888 static if(is(typeof(method) R == return)) { 10889 static if(is(R : WebObject)) { 10890 def ~= "/"; 10891 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10892 ret ~= v ~ " " ~ def; 10893 } else { 10894 if(hasAutomaticForm) { 10895 ret ~= "GET " ~ def; 10896 ret ~= "POST " ~ def; 10897 } else { 10898 ret ~= to!string(verb) ~ " " ~ def; 10899 } 10900 } 10901 } else static assert(0); 10902 10903 return ret; 10904 } 10905 enum urlNamesForMethod = helper(); 10906 } 10907 10908 10909 enum AccessCheck { 10910 allowed, 10911 denied, 10912 nonExistant, 10913 } 10914 10915 enum Operation { 10916 show, 10917 create, 10918 replace, 10919 remove, 10920 update 10921 } 10922 10923 enum UpdateResult { 10924 accessDenied, 10925 noSuchResource, 10926 success, 10927 failure, 10928 unnecessary 10929 } 10930 10931 enum ValidationResult { 10932 valid, 10933 invalid 10934 } 10935 10936 10937 /++ 10938 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 10939 10940 WARNING: this is not stable. 10941 +/ 10942 class RestObject(CRTP) : WebObject { 10943 10944 import arsd.dom; 10945 import arsd.jsvar; 10946 10947 /// Prepare the object to be shown. 10948 void show() {} 10949 /// ditto 10950 void show(string urlId) { 10951 load(urlId); 10952 show(); 10953 } 10954 10955 /// Override this to provide access control to this object. 10956 AccessCheck accessCheck(string urlId, Operation operation) { 10957 return AccessCheck.allowed; 10958 } 10959 10960 ValidationResult validate() { 10961 // FIXME 10962 return ValidationResult.valid; 10963 } 10964 10965 string getUrlSlug() { 10966 import std.conv; 10967 static if(is(typeof(CRTP.id))) 10968 return to!string((cast(CRTP) this).id); 10969 else 10970 return null; 10971 } 10972 10973 // The functions with more arguments are the low-level ones, 10974 // they forward to the ones with fewer arguments by default. 10975 10976 // POST on a parent collection - this is called from a collection class after the members are updated 10977 /++ 10978 Given a populated object, this creates a new entry. Returns the url identifier 10979 of the new object. 10980 +/ 10981 string create(scope void delegate() applyChanges) { 10982 applyChanges(); 10983 save(); 10984 return getUrlSlug(); 10985 } 10986 10987 void replace() { 10988 save(); 10989 } 10990 void replace(string urlId, scope void delegate() applyChanges) { 10991 load(urlId); 10992 applyChanges(); 10993 replace(); 10994 } 10995 10996 void update(string[] fieldList) { 10997 save(); 10998 } 10999 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 11000 load(urlId); 11001 applyChanges(); 11002 update(fieldList); 11003 } 11004 11005 void remove() {} 11006 11007 void remove(string urlId) { 11008 load(urlId); 11009 remove(); 11010 } 11011 11012 abstract void load(string urlId); 11013 abstract void save(); 11014 11015 Element toHtml(Presenter)(Presenter presenter) { 11016 import arsd.dom; 11017 import std.conv; 11018 auto obj = cast(CRTP) this; 11019 auto div = Element.make("div"); 11020 div.addClass("Dclass_" ~ CRTP.stringof); 11021 div.dataset.url = getUrlSlug(); 11022 bool first = true; 11023 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11024 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11025 if(!first) div.addChild("br"); else first = false; 11026 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 11027 } 11028 return div; 11029 } 11030 11031 var toJson() { 11032 import arsd.jsvar; 11033 var v = var.emptyObject(); 11034 auto obj = cast(CRTP) this; 11035 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 11036 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11037 v[memberName] = __traits(getMember, obj, memberName); 11038 } 11039 return v; 11040 } 11041 11042 /+ 11043 auto structOf(this This) { 11044 11045 } 11046 +/ 11047 } 11048 11049 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 11050 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 11051 11052 /++ 11053 Base class for REST collections. 11054 +/ 11055 class CollectionOf(Obj) : RestObject!(CollectionOf) { 11056 /// You might subclass this and use the cgi object's query params 11057 /// to implement a search filter, for example. 11058 /// 11059 /// FIXME: design a way to auto-generate that form 11060 /// (other than using the WebObject thing above lol 11061 // it'll prolly just be some searchParams UDA or maybe an enum. 11062 // 11063 // pagination too perhaps. 11064 // 11065 // and sorting too 11066 IndexResult index() { return IndexResult.init; } 11067 11068 string[] sortableFields() { return null; } 11069 string[] searchableFields() { return null; } 11070 11071 struct IndexResult { 11072 Obj[] results; 11073 11074 string[] sortableFields; 11075 11076 string previousPageIdentifier; 11077 string nextPageIdentifier; 11078 string firstPageIdentifier; 11079 string lastPageIdentifier; 11080 11081 int numberOfPages; 11082 } 11083 11084 override string create(scope void delegate() applyChanges) { assert(0); } 11085 override void load(string urlId) { assert(0); } 11086 override void save() { assert(0); } 11087 override void show() { 11088 index(); 11089 } 11090 override void show(string urlId) { 11091 show(); 11092 } 11093 11094 /// Proxy POST requests (create calls) to the child collection 11095 alias PostProxy = Obj; 11096 } 11097 11098 /++ 11099 Serves a REST object, similar to a Ruby on Rails resource. 11100 11101 You put data members in your class. cgi.d will automatically make something out of those. 11102 11103 It will call your constructor with the ID from the URL. This may be null. 11104 It will then populate the data members from the request. 11105 It will then call a method, if present, telling what happened. You don't need to write these! 11106 It finally returns a reply. 11107 11108 Your methods are passed a list of fields it actually set. 11109 11110 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 11111 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 11112 with relative linking. But meh.) 11113 11114 GET /items -> index. all values not set. 11115 GET /items/id -> get. only ID will be set, other params ignored. 11116 POST /items -> create. values set as given 11117 PUT /items/id -> replace. values set as given 11118 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 11119 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 11120 PATCH /items/id -> update. values set as given, list of changed fields passed 11121 or POST /items/id with cgi.post["_method"] == "PATCH" 11122 DELETE /items/id -> destroy. only ID guaranteed to be set 11123 or POST /items/id with cgi.post["_method"] == "DELETE" 11124 11125 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 11126 redirect you away from it. 11127 11128 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 11129 11130 I will also let you change the default, if you must. 11131 11132 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 11133 11134 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 11135 that follow the same thing. They may be individual resources or collections themselves. 11136 11137 Your class is expected to have at least the following methods: 11138 11139 FIXME: i kinda wanna add a routes object to the initialize call 11140 11141 create 11142 Create returns the new address on success, some code on failure. 11143 show 11144 index 11145 update 11146 remove 11147 11148 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 11149 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 11150 11151 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 11152 11153 NOT IMPLEMENTED 11154 11155 11156 Really, a collection is a resource with a bunch of subresources. 11157 11158 GET /items 11159 index because it is GET on the top resource 11160 11161 GET /items/foo 11162 item but different than items? 11163 11164 class Items { 11165 11166 } 11167 11168 ... but meh, a collection can be automated. not worth making it 11169 a separate thing, let's look at a real example. Users has many 11170 items and a virtual one, /users/current. 11171 11172 the individual users have properties and two sub-resources: 11173 session, which is just one, and comments, a collection. 11174 11175 class User : RestObject!() { // no parent 11176 int id; 11177 string name; 11178 11179 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 11180 // but you can override them to do it differently. 11181 11182 // any member which is of type RestObject can be linked automatically via href btw. 11183 11184 void show() {} 11185 void show(string urlId) {} // automated! GET of this specific thing 11186 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 11187 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 11188 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 11189 void remove(string urlId) {} // DELETE 11190 11191 void load(string urlId) {} // the default implementation of show() populates the id, then 11192 11193 this() {} 11194 11195 mixin Subresource!Session; 11196 mixin Subresource!Comment; 11197 } 11198 11199 class Session : RestObject!() { 11200 // the parent object may not be fully constructed/loaded 11201 this(User parent) {} 11202 11203 } 11204 11205 class Comment : CollectionOf!Comment { 11206 this(User parent) {} 11207 } 11208 11209 class Users : CollectionOf!User { 11210 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 11211 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 11212 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 11213 } 11214 11215 +/ 11216 auto serveRestObject(T)(string urlPrefix) { 11217 assert(urlPrefix[0] == '/'); 11218 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 11219 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 11220 string url = cgi.pathInfo[urlPrefix.length .. $]; 11221 11222 if(url.length && url[$ - 1] == '/') { 11223 // remove the final slash... 11224 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 11225 return true; 11226 } 11227 11228 return restObjectServeHandler!T(cgi, presenter, url); 11229 } 11230 return DispatcherDefinition!internalHandler(urlPrefix, false); 11231 } 11232 11233 /+ 11234 /// Convenience method for serving a collection. It will be named the same 11235 /// as type T, just with an s at the end. If you need any further, just 11236 /// write the class yourself. 11237 auto serveRestCollectionOf(T)(string urlPrefix) { 11238 assert(urlPrefix[0] == '/'); 11239 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 11240 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 11241 } 11242 +/ 11243 11244 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 11245 string urlId = null; 11246 if(url.length && url[0] == '/') { 11247 // asking for a subobject 11248 urlId = url[1 .. $]; 11249 foreach(idx, ch; urlId) { 11250 if(ch == '/') { 11251 urlId = urlId[0 .. idx]; 11252 break; 11253 } 11254 } 11255 } 11256 11257 // FIXME handle other subresources 11258 11259 static if(is(T : CollectionOf!(C), C)) { 11260 if(urlId !is null) { 11261 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 11262 } 11263 } 11264 11265 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 11266 11267 auto obj = new T(); 11268 obj.initialize(cgi); 11269 // FIXME: populate reflection info delegates 11270 11271 11272 // FIXME: I am not happy with this. 11273 switch(urlId) { 11274 case "script.js": 11275 cgi.setResponseContentType("text/javascript"); 11276 cgi.gzipResponse = true; 11277 cgi.write(presenter.script(), true); 11278 return true; 11279 case "style.css": 11280 cgi.setResponseContentType("text/css"); 11281 cgi.gzipResponse = true; 11282 cgi.write(presenter.style(), true); 11283 return true; 11284 default: 11285 // intentionally blank 11286 } 11287 11288 11289 11290 11291 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 11292 foreach(idx, memberName; __traits(derivedMembers, Obj)) 11293 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 11294 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 11295 } 11296 } 11297 void applyChanges() { 11298 applyChangesTemplate(cgi, obj); 11299 } 11300 11301 string[] modifiedList; 11302 11303 void writeObject(bool addFormLinks) { 11304 if(cgi.request("format") == "json") { 11305 cgi.setResponseContentType("application/json"); 11306 cgi.write(obj.toJson().toString, true); 11307 } else { 11308 auto container = presenter.htmlContainer(); 11309 if(addFormLinks) { 11310 static if(is(T : CollectionOf!(C), C)) 11311 container.appendHtml(` 11312 <form> 11313 <button type="submit" name="_method" value="POST">Create New</button> 11314 </form> 11315 `); 11316 else 11317 container.appendHtml(` 11318 <a href="..">Back</a> 11319 <form> 11320 <button type="submit" name="_method" value="PATCH">Edit</button> 11321 <button type="submit" name="_method" value="DELETE">Delete</button> 11322 </form> 11323 `); 11324 } 11325 container.appendChild(obj.toHtml(presenter)); 11326 cgi.write(container.parentDocument.toString, true); 11327 } 11328 } 11329 11330 // FIXME: I think I need a set type in here.... 11331 // it will be nice to pass sets of members. 11332 11333 try 11334 switch(cgi.requestMethod) { 11335 case Cgi.RequestMethod.GET: 11336 // I could prolly use template this parameters in the implementation above for some reflection stuff. 11337 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 11338 11339 // automatic forms here for usable basic auto site from browser. 11340 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 11341 switch(cgi.request("_method", "GET")) { 11342 case "GET": 11343 static if(is(T : CollectionOf!(C), C)) { 11344 auto results = obj.index(); 11345 if(cgi.request("format", "html") == "html") { 11346 auto container = presenter.htmlContainer(); 11347 auto html = presenter.formatReturnValueAsHtml(results.results); 11348 container.appendHtml(` 11349 <form> 11350 <button type="submit" name="_method" value="POST">Create New</button> 11351 </form> 11352 `); 11353 11354 container.appendChild(html); 11355 cgi.write(container.parentDocument.toString, true); 11356 } else { 11357 cgi.setResponseContentType("application/json"); 11358 import arsd.jsvar; 11359 var json = var.emptyArray; 11360 foreach(r; results.results) { 11361 var o = var.emptyObject; 11362 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 11363 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 11364 o[memberName] = __traits(getMember, r, memberName); 11365 } 11366 11367 json ~= o; 11368 } 11369 cgi.write(json.toJson(), true); 11370 } 11371 } else { 11372 obj.show(urlId); 11373 writeObject(true); 11374 } 11375 break; 11376 case "PATCH": 11377 obj.load(urlId); 11378 goto case; 11379 case "PUT": 11380 case "POST": 11381 // an editing form for the object 11382 auto container = presenter.htmlContainer(); 11383 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11384 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 11385 } else { 11386 auto form = presenter.createAutomaticFormForObject(obj); 11387 } 11388 form.attrs.method = "POST"; 11389 form.setValue("_method", cgi.request("_method", "GET")); 11390 container.appendChild(form); 11391 cgi.write(container.parentDocument.toString(), true); 11392 break; 11393 case "DELETE": 11394 // FIXME: a delete form for the object (can be phrased "are you sure?") 11395 auto container = presenter.htmlContainer(); 11396 container.appendHtml(` 11397 <form method="POST"> 11398 Are you sure you want to delete this item? 11399 <input type="hidden" name="_method" value="DELETE" /> 11400 <input type="submit" value="Yes, Delete It" /> 11401 </form> 11402 11403 `); 11404 cgi.write(container.parentDocument.toString(), true); 11405 break; 11406 default: 11407 cgi.write("bad method\n", true); 11408 } 11409 break; 11410 case Cgi.RequestMethod.POST: 11411 // this is to allow compatibility with HTML forms 11412 switch(cgi.request("_method", "POST")) { 11413 case "PUT": 11414 goto PUT; 11415 case "PATCH": 11416 goto PATCH; 11417 case "DELETE": 11418 goto DELETE; 11419 case "POST": 11420 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 11421 auto p = new obj.PostProxy(); 11422 void specialApplyChanges() { 11423 applyChangesTemplate(cgi, p); 11424 } 11425 string n = p.create(&specialApplyChanges); 11426 } else { 11427 string n = obj.create(&applyChanges); 11428 } 11429 11430 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 11431 cgi.setResponseLocation(newUrl); 11432 cgi.setResponseStatus("201 Created"); 11433 cgi.write(`The object has been created.`); 11434 break; 11435 default: 11436 cgi.write("bad method\n", true); 11437 } 11438 // FIXME this should be valid on the collection, but not the child.... 11439 // 303 See Other 11440 break; 11441 case Cgi.RequestMethod.PUT: 11442 PUT: 11443 obj.replace(urlId, &applyChanges); 11444 writeObject(false); 11445 break; 11446 case Cgi.RequestMethod.PATCH: 11447 PATCH: 11448 obj.update(urlId, &applyChanges, modifiedList); 11449 writeObject(false); 11450 break; 11451 case Cgi.RequestMethod.DELETE: 11452 DELETE: 11453 obj.remove(urlId); 11454 cgi.setResponseStatus("204 No Content"); 11455 break; 11456 default: 11457 // FIXME: OPTIONS, HEAD 11458 } 11459 catch(Throwable t) { 11460 presenter.presentExceptionAsHtml(cgi, t); 11461 } 11462 11463 return true; 11464 } 11465 11466 /+ 11467 struct SetOfFields(T) { 11468 private void[0][string] storage; 11469 void set(string what) { 11470 //storage[what] = 11471 } 11472 void unset(string what) {} 11473 void setAll() {} 11474 void unsetAll() {} 11475 bool isPresent(string what) { return false; } 11476 } 11477 +/ 11478 11479 /+ 11480 enum readonly; 11481 enum hideonindex; 11482 +/ 11483 11484 /++ 11485 Returns true if I recommend gzipping content of this type. You might 11486 want to call it from your Presenter classes before calling cgi.write. 11487 11488 --- 11489 cgi.setResponseContentType(yourContentType); 11490 cgi.gzipResponse = gzipRecommendedForContentType(yourContentType); 11491 cgi.write(yourData, true); 11492 --- 11493 11494 This is used internally by [serveStaticFile], [serveStaticFileDirectory], [serveStaticData], and maybe others I forgot to update this doc about. 11495 11496 11497 The implementation considers text content to be recommended to gzip. This may change, but it seems reasonable enough for now. 11498 11499 History: 11500 Added January 28, 2023 (dub v11.0) 11501 +/ 11502 bool gzipRecommendedForContentType(string contentType) { 11503 if(contentType.startsWith("text/")) 11504 return true; 11505 if(contentType.startsWith("application/javascript")) 11506 return true; 11507 11508 return false; 11509 } 11510 11511 /++ 11512 Serves a static file. To be used with [dispatcher]. 11513 11514 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 11515 +/ 11516 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 11517 // https://baus.net/on-tcp_cork/ 11518 // man 2 sendfile 11519 assert(urlPrefix[0] == '/'); 11520 if(filename is null) 11521 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 11522 if(contentType is null) { 11523 contentType = contentTypeFromFileExtension(filename); 11524 } 11525 11526 static struct DispatcherDetails { 11527 string filename; 11528 string contentType; 11529 } 11530 11531 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11532 if(details.contentType.indexOf("image/") == 0 || details.contentType.indexOf("audio/") == 0) 11533 cgi.setCache(true); 11534 cgi.setResponseContentType(details.contentType); 11535 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11536 cgi.write(std.file.read(details.filename), true); 11537 return true; 11538 } 11539 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 11540 } 11541 11542 /++ 11543 Serves static data. To be used with [dispatcher]. 11544 11545 History: 11546 Added October 31, 2021 11547 +/ 11548 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 11549 assert(urlPrefix[0] == '/'); 11550 if(contentType is null) { 11551 contentType = contentTypeFromFileExtension(urlPrefix); 11552 } 11553 11554 static struct DispatcherDetails { 11555 immutable(void)[] data; 11556 string contentType; 11557 } 11558 11559 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11560 cgi.setCache(true); 11561 cgi.setResponseContentType(details.contentType); 11562 cgi.write(details.data, true); 11563 return true; 11564 } 11565 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 11566 } 11567 11568 string contentTypeFromFileExtension(string filename) { 11569 if(filename.endsWith(".png")) 11570 return "image/png"; 11571 if(filename.endsWith(".apng")) 11572 return "image/apng"; 11573 if(filename.endsWith(".svg")) 11574 return "image/svg+xml"; 11575 if(filename.endsWith(".jpg")) 11576 return "image/jpeg"; 11577 if(filename.endsWith(".html")) 11578 return "text/html"; 11579 if(filename.endsWith(".css")) 11580 return "text/css"; 11581 if(filename.endsWith(".js")) 11582 return "application/javascript"; 11583 if(filename.endsWith(".wasm")) 11584 return "application/wasm"; 11585 if(filename.endsWith(".mp3")) 11586 return "audio/mpeg"; 11587 if(filename.endsWith(".pdf")) 11588 return "application/pdf"; 11589 return null; 11590 } 11591 11592 /// This serves a directory full of static files, figuring out the content-types from file extensions. 11593 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 11594 auto serveStaticFileDirectory(string urlPrefix, string directory = null, bool recursive = false) { 11595 assert(urlPrefix[0] == '/'); 11596 assert(urlPrefix[$-1] == '/'); 11597 11598 static struct DispatcherDetails { 11599 string directory; 11600 bool recursive; 11601 } 11602 11603 if(directory is null) 11604 directory = urlPrefix[1 .. $]; 11605 11606 if(directory.length == 0) 11607 directory = "./"; 11608 11609 assert(directory[$-1] == '/'); 11610 11611 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11612 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11613 11614 if(details.recursive) { 11615 // never allow a backslash since it isn't in a typical url anyway and makes the following checks easier 11616 if(file.indexOf("\\") != -1) 11617 return false; 11618 11619 import std.path; 11620 11621 file = std.path.buildNormalizedPath(file); 11622 enum upOneDir = ".." ~ std.path.dirSeparator; 11623 11624 // also no point doing any kind of up directory things since that makes it more likely to break out of the parent 11625 if(file == ".." || file.startsWith(upOneDir)) 11626 return false; 11627 if(std.path.isAbsolute(file)) 11628 return false; 11629 11630 // FIXME: if it has slashes and stuff, should we redirect to the canonical resource? or what? 11631 11632 // once it passes these filters it is probably ok. 11633 } else { 11634 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11635 return false; 11636 } 11637 11638 auto contentType = contentTypeFromFileExtension(file); 11639 11640 auto fn = details.directory ~ file; 11641 if(std.file.exists(fn)) { 11642 //if(contentType.indexOf("image/") == 0) 11643 //cgi.setCache(true); 11644 //else if(contentType.indexOf("audio/") == 0) 11645 cgi.setCache(true); 11646 cgi.setResponseContentType(contentType); 11647 cgi.gzipResponse = gzipRecommendedForContentType(contentType); 11648 cgi.write(std.file.read(fn), true); 11649 return true; 11650 } else { 11651 return false; 11652 } 11653 } 11654 11655 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory, recursive)); 11656 } 11657 11658 /++ 11659 Redirects one url to another 11660 11661 See_Also: [dispatcher], [serveStaticFile] 11662 +/ 11663 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11664 assert(urlPrefix[0] == '/'); 11665 static struct DispatcherDetails { 11666 string redirectTo; 11667 string code; 11668 } 11669 11670 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11671 cgi.setResponseLocation(details.redirectTo, true, details.code); 11672 return true; 11673 } 11674 11675 11676 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11677 } 11678 11679 /// Used exclusively with `dispatchTo` 11680 struct DispatcherData(Presenter) { 11681 Cgi cgi; /// You can use this cgi object. 11682 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11683 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11684 } 11685 11686 /++ 11687 Dispatches the URL to a specific function. 11688 +/ 11689 auto handleWith(alias handler)(string urlPrefix) { 11690 // cuz I'm too lazy to do it better right now 11691 static class Hack : WebObject { 11692 static import std.traits; 11693 @UrlName("") 11694 auto handle(std.traits.Parameters!handler args) { 11695 return handler(args); 11696 } 11697 } 11698 11699 return urlPrefix.serveApiInternal!Hack; 11700 } 11701 11702 /++ 11703 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11704 11705 --- 11706 bool other(DD)(DD dd) { 11707 return dd.dispatcher!( 11708 "/whatever".serveRedirect("/success"), 11709 "/api/".serveApi!MyClass 11710 ); 11711 } 11712 --- 11713 11714 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11715 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11716 Or, of course, you could just use the exact type in your own code. 11717 11718 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11719 good job. 11720 11721 11722 +/ 11723 auto dispatchTo(alias handler)(string urlPrefix) { 11724 assert(urlPrefix[0] == '/'); 11725 assert(urlPrefix[$-1] != '/'); 11726 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11727 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11728 } 11729 11730 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11731 } 11732 11733 /++ 11734 See [serveStaticFile] if you want to serve a file off disk. 11735 11736 History: 11737 Added January 28, 2023 (dub v11.0) 11738 +/ 11739 auto serveStaticData(string urlPrefix, immutable(ubyte)[] data, string contentType, string filenameToSuggestAsDownload = null) { 11740 assert(urlPrefix[0] == '/'); 11741 11742 static struct DispatcherDetails { 11743 immutable(ubyte)[] data; 11744 string contentType; 11745 string filenameToSuggestAsDownload; 11746 } 11747 11748 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11749 cgi.setCache(true); 11750 cgi.setResponseContentType(details.contentType); 11751 if(details.filenameToSuggestAsDownload.length) 11752 cgi.header("Content-Disposition: attachment; filename=\""~details.filenameToSuggestAsDownload~"\""); 11753 cgi.gzipResponse = gzipRecommendedForContentType(details.contentType); 11754 cgi.write(details.data, true); 11755 return true; 11756 } 11757 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType, filenameToSuggestAsDownload)); 11758 } 11759 11760 /++ 11761 Placeholder for use with [dispatchSubsection]'s `NewPresenter` argument to indicate you want to keep the parent's presenter. 11762 11763 History: 11764 Added January 28, 2023 (dub v11.0) 11765 +/ 11766 alias KeepExistingPresenter = typeof(null); 11767 11768 /++ 11769 For use with [dispatchSubsection]. Calls your filter with the request and if your filter returns false, 11770 this issues the given errorCode and stops processing. 11771 11772 --- 11773 bool hasAdminPermissions(Cgi cgi) { 11774 return true; 11775 } 11776 11777 mixin DispatcherMain!( 11778 "/admin".dispatchSubsection!( 11779 passFilterOrIssueError!(hasAdminPermissions, 403), 11780 KeepExistingPresenter, 11781 "/".serveApi!AdminFunctions 11782 ) 11783 ); 11784 --- 11785 11786 History: 11787 Added January 28, 2023 (dub v11.0) 11788 +/ 11789 template passFilterOrIssueError(alias filter, int errorCode) { 11790 bool passFilterOrIssueError(DispatcherDetails)(DispatcherDetails dd) { 11791 if(filter(dd.cgi)) 11792 return true; 11793 dd.presenter.renderBasicError(dd.cgi, errorCode); 11794 return false; 11795 } 11796 } 11797 11798 /++ 11799 Allows for a subsection of your dispatched urls to be passed through other a pre-request filter, optionally pick up an new presenter class, 11800 and then be dispatched to their own handlers. 11801 11802 --- 11803 /+ 11804 // a long-form filter function 11805 bool permissionCheck(DispatcherData)(DispatcherData dd) { 11806 // you are permitted to call mutable methods on the Cgi object 11807 // Note if you use a Cgi subclass, you can try dynamic casting it back to your custom type to attach per-request data 11808 // though much of the request is immutable so there's only so much you're allowed to do to modify it. 11809 11810 if(checkPermissionOnRequest(dd.cgi)) { 11811 return true; // OK, allow processing to continue 11812 } else { 11813 dd.presenter.renderBasicError(dd.cgi, 403); // reply forbidden to the requester 11814 return false; // and stop further processing into this subsection 11815 } 11816 } 11817 +/ 11818 11819 // but you can also do short-form filters: 11820 11821 bool permissionCheck(Cgi cgi) { 11822 return ("ok" in cgi.get) !is null; 11823 } 11824 11825 // handler for the subsection 11826 class AdminClass : WebObject { 11827 int foo() { return 5; } 11828 } 11829 11830 // handler for the main site 11831 class TheMainSite : WebObject {} 11832 11833 mixin DispatcherMain!( 11834 "/admin".dispatchSubsection!( 11835 // converts our short-form filter into a long-form filter 11836 passFilterOrIssueError!(permissionCheck, 403), 11837 // can use a new presenter if wanted for the subsection 11838 KeepExistingPresenter, 11839 // and then provide child route dispatchers 11840 "/".serveApi!AdminClass 11841 ), 11842 // and back to the top level 11843 "/".serveApi!TheMainSite 11844 ); 11845 --- 11846 11847 Note you can encapsulate sections in files like this: 11848 11849 --- 11850 auto adminDispatcher(string urlPrefix) { 11851 return urlPrefix.dispatchSubsection!( 11852 .... 11853 ); 11854 } 11855 11856 mixin DispatcherMain!( 11857 "/admin".adminDispatcher, 11858 // and so on 11859 ) 11860 --- 11861 11862 If you want no filter, you can pass `(cgi) => true` as the filter to approve all requests. 11863 11864 If you want to keep the same presenter as the parent, use [KeepExistingPresenter] as the presenter argument. 11865 11866 11867 History: 11868 Added January 28, 2023 (dub v11.0) 11869 +/ 11870 auto dispatchSubsection(alias PreRequestFilter, NewPresenter, definitions...)(string urlPrefix) { 11871 assert(urlPrefix[0] == '/'); 11872 assert(urlPrefix[$-1] != '/'); 11873 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11874 static if(!is(PreRequestFilter == typeof(null))) { 11875 if(!PreRequestFilter(DispatcherData!Presenter(cgi, presenter, urlPrefix.length))) 11876 return true; // we handled it by rejecting it 11877 } 11878 11879 static if(is(NewPresenter == Presenter) || is(NewPresenter == typeof(null))) { 11880 return dispatcher!definitions(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11881 } else { 11882 auto newPresenter = new NewPresenter(); 11883 return dispatcher!(definitions(DispatcherData!NewPresenter(cgi, presenter, urlPrefix.length))); 11884 } 11885 } 11886 11887 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11888 } 11889 11890 /++ 11891 A URL dispatcher. 11892 11893 --- 11894 if(cgi.dispatcher!( 11895 "/api/".serveApi!MyApiClass, 11896 "/objects/lol".serveRestObject!MyRestObject, 11897 "/file.js".serveStaticFile, 11898 "/admin/".dispatchTo!adminHandler 11899 )) return; 11900 --- 11901 11902 11903 You define a series of url prefixes followed by handlers. 11904 11905 You may want to do different pre- and post- processing there, for example, 11906 an authorization check and different page layout. You can use different 11907 presenters and different function chains. See [dispatchSubsection] for details. 11908 11909 [dispatchTo] will send the request to another function for handling. 11910 +/ 11911 template dispatcher(definitions...) { 11912 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11913 static if(is(Presenter == typeof(null))) { 11914 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11915 auto presenter = new GenericWebPresenter(); 11916 } else 11917 alias presenter = presenterArg; 11918 11919 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 11920 } 11921 11922 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 11923 // I can prolly make this more efficient later but meh. 11924 foreach(definition; definitions) { 11925 if(definition.rejectFurther) { 11926 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 11927 auto ret = definition.handler( 11928 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11929 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11930 if(ret) 11931 return true; 11932 } 11933 } else if( 11934 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 11935 // cgi.d dispatcher urls must be complete or have a /; 11936 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 11937 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 11938 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 11939 ) { 11940 auto ret = definition.handler( 11941 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11942 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11943 if(ret) 11944 return true; 11945 } 11946 } 11947 return false; 11948 } 11949 } 11950 11951 }); 11952 11953 private struct StackBuffer { 11954 char[1024] initial = void; 11955 char[] buffer; 11956 size_t position; 11957 11958 this(int a) { 11959 buffer = initial[]; 11960 position = 0; 11961 } 11962 11963 void add(in char[] what) { 11964 if(position + what.length > buffer.length) 11965 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 11966 buffer[position .. position + what.length] = what[]; 11967 position += what.length; 11968 } 11969 11970 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 11971 add(w1); 11972 add(w2); 11973 add(w3); 11974 } 11975 11976 void add(long v) { 11977 char[16] buffer = void; 11978 auto pos = buffer.length; 11979 bool negative; 11980 if(v < 0) { 11981 negative = true; 11982 v = -v; 11983 } 11984 do { 11985 buffer[--pos] = cast(char) (v % 10 + '0'); 11986 v /= 10; 11987 } while(v); 11988 11989 if(negative) 11990 buffer[--pos] = '-'; 11991 11992 auto res = buffer[pos .. $]; 11993 11994 add(res[]); 11995 } 11996 11997 char[] get() @nogc { 11998 return buffer[0 .. position]; 11999 } 12000 } 12001 12002 // duplicated in http2.d 12003 private static string getHttpCodeText(int code) pure nothrow @nogc { 12004 switch(code) { 12005 case 200: return "200 OK"; 12006 case 201: return "201 Created"; 12007 case 202: return "202 Accepted"; 12008 case 203: return "203 Non-Authoritative Information"; 12009 case 204: return "204 No Content"; 12010 case 205: return "205 Reset Content"; 12011 case 206: return "206 Partial Content"; 12012 // 12013 case 300: return "300 Multiple Choices"; 12014 case 301: return "301 Moved Permanently"; 12015 case 302: return "302 Found"; 12016 case 303: return "303 See Other"; 12017 case 304: return "304 Not Modified"; 12018 case 305: return "305 Use Proxy"; 12019 case 307: return "307 Temporary Redirect"; 12020 case 308: return "308 Permanent Redirect"; 12021 12022 // 12023 case 400: return "400 Bad Request"; 12024 case 401: return "401 Unauthorized"; 12025 case 402: return "402 Payment Required"; 12026 case 403: return "403 Forbidden"; 12027 case 404: return "404 Not Found"; 12028 case 405: return "405 Method Not Allowed"; 12029 case 406: return "406 Not Acceptable"; 12030 case 407: return "407 Proxy Authentication Required"; 12031 case 408: return "408 Request Timeout"; 12032 case 409: return "409 Conflict"; 12033 case 410: return "410 Gone"; 12034 case 411: return "411 Length Required"; 12035 case 412: return "412 Precondition Failed"; 12036 case 413: return "413 Payload Too Large"; 12037 case 414: return "414 URI Too Long"; 12038 case 415: return "415 Unsupported Media Type"; 12039 case 416: return "416 Range Not Satisfiable"; 12040 case 417: return "417 Expectation Failed"; 12041 case 418: return "418 I'm a teapot"; 12042 case 421: return "421 Misdirected Request"; 12043 case 422: return "422 Unprocessable Entity (WebDAV)"; 12044 case 423: return "423 Locked (WebDAV)"; 12045 case 424: return "424 Failed Dependency (WebDAV)"; 12046 case 425: return "425 Too Early"; 12047 case 426: return "426 Upgrade Required"; 12048 case 428: return "428 Precondition Required"; 12049 case 431: return "431 Request Header Fields Too Large"; 12050 case 451: return "451 Unavailable For Legal Reasons"; 12051 12052 case 500: return "500 Internal Server Error"; 12053 case 501: return "501 Not Implemented"; 12054 case 502: return "502 Bad Gateway"; 12055 case 503: return "503 Service Unavailable"; 12056 case 504: return "504 Gateway Timeout"; 12057 case 505: return "505 HTTP Version Not Supported"; 12058 case 506: return "506 Variant Also Negotiates"; 12059 case 507: return "507 Insufficient Storage (WebDAV)"; 12060 case 508: return "508 Loop Detected (WebDAV)"; 12061 case 510: return "510 Not Extended"; 12062 case 511: return "511 Network Authentication Required"; 12063 // 12064 default: assert(0, "Unsupported http code"); 12065 } 12066 } 12067 12068 12069 /+ 12070 /++ 12071 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 12072 12073 It relies on jsvar.d and dom.d. 12074 12075 12076 You can get javascript out of it to call. The generated functions need to look 12077 like 12078 12079 function name(a,b,c,d,e) { 12080 return _call("name", {"realName":a,"sds":b}); 12081 } 12082 12083 And _call returns an object you can call or set up or whatever. 12084 +/ 12085 bool apiDispatcher()(Cgi cgi) { 12086 import arsd.jsvar; 12087 import arsd.dom; 12088 } 12089 +/ 12090 version(linux) 12091 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 12092 /* 12093 Copyright: Adam D. Ruppe, 2008 - 2023 12094 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 12095 Authors: Adam D. Ruppe 12096 12097 Copyright Adam D. Ruppe 2008 - 2023. 12098 Distributed under the Boost Software License, Version 1.0. 12099 (See accompanying file LICENSE_1_0.txt or copy at 12100 http://www.boost.org/LICENSE_1_0.txt) 12101 */