1 // FIXME: if an exception is thrown, we shouldn't necessarily cache... 2 // FIXME: there's some annoying duplication of code in the various versioned mains 3 4 // add the Range header in there too. should return 206 5 6 // FIXME: cgi per-request arena allocator 7 8 // i need to add a bunch of type templates for validations... mayne @NotNull or NotNull! 9 10 // FIXME: I might make a cgi proxy class which can change things; the underlying one is still immutable 11 // but the later one can edit and simplify the api. You'd have to use the subclass tho! 12 13 /* 14 void foo(int f, @("test") string s) {} 15 16 void main() { 17 static if(is(typeof(foo) Params == __parameters)) 18 //pragma(msg, __traits(getAttributes, Params[0])); 19 pragma(msg, __traits(getAttributes, Params[1..2])); 20 else 21 pragma(msg, "fail"); 22 } 23 */ 24 25 // Note: spawn-fcgi can help with fastcgi on nginx 26 27 // FIXME: to do: add openssl optionally 28 // make sure embedded_httpd doesn't send two answers if one writes() then dies 29 30 // future direction: websocket as a separate process that you can sendfile to for an async passoff of those long-lived connections 31 32 /* 33 Session manager process: it spawns a new process, passing a 34 command line argument, to just be a little key/value store 35 of some serializable struct. On Windows, it CreateProcess. 36 On Linux, it can just fork or maybe fork/exec. The session 37 key is in a cookie. 38 39 Server-side event process: spawns an async manager. You can 40 push stuff out to channel ids and the clients listen to it. 41 42 websocket process: spawns an async handler. They can talk to 43 each other or get info from a cgi request. 44 45 Tempting to put web.d 2.0 in here. It would: 46 * map urls and form generation to functions 47 * have data presentation magic 48 * do the skeleton stuff like 1.0 49 * auto-cache generated stuff in files (at least if pure?) 50 * introspect functions in json for consumers 51 52 53 https://linux.die.net/man/3/posix_spawn 54 */ 55 56 /++ 57 Provides a uniform server-side API for CGI, FastCGI, SCGI, and HTTP web applications. 58 59 --- 60 import arsd.cgi; 61 62 // Instead of writing your own main(), you should write a function 63 // that takes a Cgi param, and use mixin GenericMain 64 // for maximum compatibility with different web servers. 65 void hello(Cgi cgi) { 66 cgi.setResponseContentType("text/plain"); 67 68 if("name" in cgi.get) 69 cgi.write("Hello, " ~ cgi.get["name"]); 70 else 71 cgi.write("Hello, world!"); 72 } 73 74 mixin GenericMain!hello; 75 --- 76 77 Test on console (works in any interface mode): 78 $(CONSOLE 79 $ ./cgi_hello GET / name=whatever 80 ) 81 82 If using http version (default on `dub` builds, or on custom builds when passing `-version=embedded_httpd` to dmd): 83 $(CONSOLE 84 $ ./cgi_hello --port 8080 85 # now you can go to http://localhost:8080/?name=whatever 86 ) 87 88 Please note: the default port for http is 8085 and for cgi is 4000. I recommend you set your own by the command line argument in a startup script instead of relying on any hard coded defaults. It is possible though to hard code your own with [RequestServer]. 89 90 91 Compile_versions: 92 93 If you are using `dub`, use: 94 95 ```sdlang 96 subConfiguration "arsd-official:cgi" "VALUE_HERE" 97 ``` 98 99 or to dub.json: 100 101 ```json 102 "subConfigurations": {"arsd-official:cgi": "VALUE_HERE"} 103 ``` 104 105 to change versions. The possible options for `VALUE_HERE` are: 106 107 $(LIST 108 * `embedded_httpd` for the embedded httpd version (built-in web server). This is the default. 109 * `cgi` for traditional cgi binaries. 110 * `fastcgi` for FastCGI builds. 111 * `scgi` for SCGI builds. 112 * `stdio_http` for speaking raw http over stdin and stdout. See [RequestServer.serveSingleHttpConnectionOnStdio] for more information. 113 ) 114 115 With dmd, use: 116 117 $(TABLE_ROWS 118 119 * + Interfaces 120 + (mutually exclusive) 121 122 * - `-version=plain_cgi` 123 - The default building the module alone without dub - a traditional, plain CGI executable will be generated. 124 * - `-version=embedded_httpd` 125 - A HTTP server will be embedded in the generated executable. This is default when building with dub. 126 * - `-version=fastcgi` 127 - A FastCGI executable will be generated. 128 * - `-version=scgi` 129 - A SCGI (SimpleCGI) executable will be generated. 130 131 * - `-version=embedded_httpd_threads` 132 - The embedded HTTP server will use a single process with a thread pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 133 * - `-version=embedded_httpd_processes` 134 - The embedded HTTP server will use a prefork style process pool. (use instead of plain `embedded_httpd` if you want this specific implementation) 135 * - `-version=embedded_httpd_processes_accept_after_fork` 136 - It will call accept() in each child process, after forking. This is currently the only option, though I am experimenting with other ideas. You probably should NOT specify this right now. 137 * - `-version=stdio_http` 138 - The embedded HTTP server will be spoken over stdin and stdout. 139 140 * + Tweaks 141 + (can be used together with others) 142 143 * - `-version=cgi_with_websocket` 144 - The CGI class has websocket server support. 145 146 * - `-version=with_openssl` 147 - not currently used 148 * - `-version=cgi_embedded_sessions` 149 - The session server will be embedded in the cgi.d server process 150 * - `-version=cgi_session_server_process` 151 - The session will be provided in a separate process, provided by cgi.d. 152 ) 153 154 Compile_and_run: 155 156 For CGI, `dmd yourfile.d cgi.d` then put the executable in your cgi-bin directory. 157 158 For FastCGI: `dmd yourfile.d cgi.d -version=fastcgi` and run it. spawn-fcgi helps on nginx. You can put the file in the directory for Apache. On IIS, run it with a port on the command line (this causes it to call FCGX_OpenSocket, which can work on nginx too). 159 160 For SCGI: `dmd yourfile.d cgi.d -version=scgi` and run the executable, providing a port number on the command line. 161 162 For an embedded HTTP server, run `dmd yourfile.d cgi.d -version=embedded_httpd` and run the generated program. It listens on port 8085 by default. You can change this on the command line with the --port option when running your program. 163 164 You can also simulate a request by passing parameters on the command line, like: 165 166 $(CONSOLE 167 ./yourprogram GET / name=adr 168 ) 169 170 And it will print the result to stdout. 171 172 CGI_Setup_tips: 173 174 On Apache, you may do `SetHandler cgi-script` in your `.htaccess` file. 175 176 Integration_tips: 177 178 cgi.d works well with dom.d for generating html. You may also use web.d for other utilities and automatic api wrapping. 179 180 dom.d usage: 181 182 --- 183 import arsd.cgi; 184 import arsd.dom; 185 186 void hello_dom(Cgi cgi) { 187 auto document = new Document(); 188 189 static import std.file; 190 // parse the file in strict mode, requiring it to be well-formed UTF-8 XHTML 191 // (You'll appreciate this if you've ever had to deal with a missing </div> 192 // or something in a php or erb template before that would randomly mess up 193 // the output in your browser. Just check it and throw an exception early!) 194 // 195 // You could also hard-code a template or load one at compile time with an 196 // import expression, but you might appreciate making it a regular file 197 // because that means it can be more easily edited by the frontend team and 198 // they can see their changes without needing to recompile the program. 199 // 200 // Note on CTFE: if you do choose to load a static file at compile time, 201 // you *can* parse it in CTFE using enum, which will cause it to throw at 202 // compile time, which is kinda cool too. Be careful in modifying that document, 203 // though, as it will be a static instance. You might want to clone on on demand, 204 // or perhaps modify it lazily as you print it out. (Try element.tree, it returns 205 // a range of elements which you could send through std.algorithm functions. But 206 // since my selector implementation doesn't work on that level yet, you'll find that 207 // harder to use. Of course, you could make a static list of matching elements and 208 // then use a simple e is e2 predicate... :) ) 209 document.parseUtf8(std.file.read("your_template.html"), true, true); 210 211 // fill in data using DOM functions, so placing it is in the hands of HTML 212 // and it will be properly encoded as text too. 213 // 214 // Plain html templates can't run server side logic, but I think that's a 215 // good thing - it keeps them simple. You may choose to extend the html, 216 // but I think it is best to try to stick to standard elements and fill them 217 // in with requested data with IDs or class names. A further benefit of 218 // this is the designer can also highlight data based on sources in the CSS. 219 // 220 // However, all of dom.d is available, so you can format your data however 221 // you like. You can do partial templates with innerHTML too, or perhaps better, 222 // injecting cloned nodes from a partial document. 223 // 224 // There's a lot of possibilities. 225 document["#name"].innerText = cgi.request("name", "default name"); 226 227 // send the document to the browser. The second argument to `cgi.write` 228 // indicates that this is all the data at once, enabling a few small 229 // optimizations. 230 cgi.write(document.toString(), true); 231 } 232 --- 233 234 Concepts: 235 Input: [Cgi.get], [Cgi.post], [Cgi.request], [Cgi.files], [Cgi.cookies], [Cgi.pathInfo], [Cgi.requestMethod], 236 and HTTP headers ([Cgi.headers], [Cgi.userAgent], [Cgi.referrer], [Cgi.accept], [Cgi.authorization], [Cgi.lastEventId]) 237 238 Output: [Cgi.write], [Cgi.header], [Cgi.setResponseStatus], [Cgi.setResponseContentType], [Cgi.gzipResponse] 239 240 Cookies: [Cgi.setCookie], [Cgi.clearCookie], [Cgi.cookie], [Cgi.cookies] 241 242 Caching: [Cgi.setResponseExpires], [Cgi.updateResponseExpires], [Cgi.setCache] 243 244 Redirections: [Cgi.setResponseLocation] 245 246 Other Information: [Cgi.remoteAddress], [Cgi.https], [Cgi.port], [Cgi.scriptName], [Cgi.requestUri], [Cgi.getCurrentCompleteUri], [Cgi.onRequestBodyDataReceived] 247 248 Overriding behavior: [Cgi.handleIncomingDataChunk], [Cgi.prepareForIncomingDataChunks], [Cgi.cleanUpPostDataState] 249 250 Installing: Apache, IIS, CGI, FastCGI, SCGI, embedded HTTPD (not recommended for production use) 251 252 Guide_for_PHP_users: 253 If you are coming from PHP, here's a quick guide to help you get started: 254 255 $(SIDE_BY_SIDE 256 $(COLUMN 257 ```php 258 <?php 259 $foo = $_POST["foo"]; 260 $bar = $_GET["bar"]; 261 $baz = $_COOKIE["baz"]; 262 263 $user_ip = $_SERVER["REMOTE_ADDR"]; 264 $host = $_SERVER["HTTP_HOST"]; 265 $path = $_SERVER["PATH_INFO"]; 266 267 setcookie("baz", "some value"); 268 269 echo "hello!"; 270 ?> 271 ``` 272 ) 273 $(COLUMN 274 --- 275 import arsd.cgi; 276 void app(Cgi cgi) { 277 string foo = cgi.post["foo"]; 278 string bar = cgi.get["bar"]; 279 string baz = cgi.cookies["baz"]; 280 281 string user_ip = cgi.remoteAddress; 282 string host = cgi.host; 283 string path = cgi.pathInfo; 284 285 cgi.setCookie("baz", "some value"); 286 287 cgi.write("hello!"); 288 } 289 290 mixin GenericMain!app 291 --- 292 ) 293 ) 294 295 $(H3 Array elements) 296 297 298 In PHP, you can give a form element a name like `"something[]"`, and then 299 `$_POST["something"]` gives an array. In D, you can use whatever name 300 you want, and access an array of values with the `cgi.getArray["name"]` and 301 `cgi.postArray["name"]` members. 302 303 $(H3 Databases) 304 305 PHP has a lot of stuff in its standard library. cgi.d doesn't include most 306 of these, but the rest of my arsd repository has much of it. For example, 307 to access a MySQL database, download `database.d` and `mysql.d` from my 308 github repo, and try this code (assuming, of course, your database is 309 set up): 310 311 --- 312 import arsd.cgi; 313 import arsd.mysql; 314 315 void app(Cgi cgi) { 316 auto database = new MySql("localhost", "username", "password", "database_name"); 317 foreach(row; mysql.query("SELECT count(id) FROM people")) 318 cgi.write(row[0] ~ " people in database"); 319 } 320 321 mixin GenericMain!app; 322 --- 323 324 Similar modules are available for PostgreSQL, Microsoft SQL Server, and SQLite databases, 325 implementing the same basic interface. 326 327 See_Also: 328 329 You may also want to see [arsd.dom], [arsd.web], and [arsd.html] for more code for making 330 web applications. 331 332 For working with json, try [arsd.jsvar]. 333 334 [arsd.database], [arsd.mysql], [arsd.postgres], [arsd.mssql], and [arsd.sqlite] can help in 335 accessing databases. 336 337 If you are looking to access a web application via HTTP, try [std.net.curl], [arsd.curl], or [arsd.http2]. 338 339 Copyright: 340 341 cgi.d copyright 2008-2022, Adam D. Ruppe. Provided under the Boost Software License. 342 343 Yes, this file is old, and yes, it is still actively maintained and used. 344 +/ 345 module arsd.cgi; 346 347 version(Demo) 348 unittest { 349 350 } 351 352 static import std.file; 353 354 // for a single thread, linear request thing, use: 355 // -version=embedded_httpd_threads -version=cgi_no_threads 356 357 version(Posix) { 358 version(CRuntime_Musl) { 359 360 } else version(minimal) { 361 362 } else { 363 version(GNU) { 364 // GDC doesn't support static foreach so I had to cheat on it :( 365 } else version(FreeBSD) { 366 // I never implemented the fancy stuff there either 367 } else { 368 version=with_breaking_cgi_features; 369 version=with_sendfd; 370 version=with_addon_servers; 371 } 372 } 373 } 374 375 version(Windows) { 376 version(minimal) { 377 378 } else { 379 // not too concerned about gdc here since the mingw version is fairly new as well 380 version=with_breaking_cgi_features; 381 } 382 } 383 384 void cloexec(int fd) { 385 version(Posix) { 386 import core.sys.posix.fcntl; 387 fcntl(fd, F_SETFD, FD_CLOEXEC); 388 } 389 } 390 391 void cloexec(Socket s) { 392 version(Posix) { 393 import core.sys.posix.fcntl; 394 fcntl(s.handle, F_SETFD, FD_CLOEXEC); 395 } 396 } 397 398 version(embedded_httpd_hybrid) { 399 version=embedded_httpd_threads; 400 version(cgi_no_fork) {} else version(Posix) 401 version=cgi_use_fork; 402 version=cgi_use_fiber; 403 } 404 405 version(cgi_use_fork) 406 enum cgi_use_fork_default = true; 407 else 408 enum cgi_use_fork_default = false; 409 410 // the servers must know about the connections to talk to them; the interfaces are vital 411 version(with_addon_servers) 412 version=with_addon_servers_connections; 413 414 version(embedded_httpd) { 415 version(linux) 416 version=embedded_httpd_processes; 417 else { 418 version=embedded_httpd_threads; 419 } 420 421 /* 422 version(with_openssl) { 423 pragma(lib, "crypto"); 424 pragma(lib, "ssl"); 425 } 426 */ 427 } 428 429 version(embedded_httpd_processes) 430 version=embedded_httpd_processes_accept_after_fork; // I am getting much better average performance on this, so just keeping it. But the other way MIGHT help keep the variation down so i wanna keep the code to play with later 431 432 version(embedded_httpd_threads) { 433 // unless the user overrides the default.. 434 version(cgi_session_server_process) 435 {} 436 else 437 version=cgi_embedded_sessions; 438 } 439 version(scgi) { 440 // unless the user overrides the default.. 441 version(cgi_session_server_process) 442 {} 443 else 444 version=cgi_embedded_sessions; 445 } 446 447 // fall back if the other is not defined so we can cleanly version it below 448 version(cgi_embedded_sessions) {} 449 else version=cgi_session_server_process; 450 451 452 version=cgi_with_websocket; 453 454 enum long defaultMaxContentLength = 5_000_000; 455 456 /* 457 458 To do a file download offer in the browser: 459 460 cgi.setResponseContentType("text/csv"); 461 cgi.header("Content-Disposition: attachment; filename=\"customers.csv\""); 462 */ 463 464 // FIXME: the location header is supposed to be an absolute url I guess. 465 466 // FIXME: would be cool to flush part of a dom document before complete 467 // somehow in here and dom.d. 468 469 470 // these are public so you can mixin GenericMain. 471 // FIXME: use a function level import instead! 472 public import std.string; 473 public import std.stdio; 474 public import std.conv; 475 import std.uri; 476 import std.uni; 477 import std.algorithm.comparison; 478 import std.algorithm.searching; 479 import std.exception; 480 import std.base64; 481 static import std.algorithm; 482 import std.datetime; 483 import std.range; 484 485 import std.process; 486 487 import std.zlib; 488 489 490 T[] consume(T)(T[] range, int count) { 491 if(count > range.length) 492 count = range.length; 493 return range[count..$]; 494 } 495 496 int locationOf(T)(T[] data, string item) { 497 const(ubyte[]) d = cast(const(ubyte[])) data; 498 const(ubyte[]) i = cast(const(ubyte[])) item; 499 500 // this is a vague sanity check to ensure we aren't getting insanely 501 // sized input that will infinite loop below. it should never happen; 502 // even huge file uploads ought to come in smaller individual pieces. 503 if(d.length > (int.max/2)) 504 throw new Exception("excessive block of input"); 505 506 for(int a = 0; a < d.length; a++) { 507 if(a + i.length > d.length) 508 return -1; 509 if(d[a..a+i.length] == i) 510 return a; 511 } 512 513 return -1; 514 } 515 516 /// If you are doing a custom cgi class, mixing this in can take care of 517 /// the required constructors for you 518 mixin template ForwardCgiConstructors() { 519 this(long maxContentLength = defaultMaxContentLength, 520 string[string] env = null, 521 const(ubyte)[] delegate() readdata = null, 522 void delegate(const(ubyte)[]) _rawDataOutput = null, 523 void delegate() _flush = null 524 ) { super(maxContentLength, env, readdata, _rawDataOutput, _flush); } 525 526 this(string[] args) { super(args); } 527 528 this( 529 BufferedInputRange inputData, 530 string address, ushort _port, 531 int pathInfoStarts = 0, 532 bool _https = false, 533 void delegate(const(ubyte)[]) _rawDataOutput = null, 534 void delegate() _flush = null, 535 // this pointer tells if the connection is supposed to be closed after we handle this 536 bool* closeConnection = null) 537 { 538 super(inputData, address, _port, pathInfoStarts, _https, _rawDataOutput, _flush, closeConnection); 539 } 540 541 this(BufferedInputRange ir, bool* closeConnection) { super(ir, closeConnection); } 542 } 543 544 /// thrown when a connection is closed remotely while we waiting on data from it 545 class ConnectionClosedException : Exception { 546 this(string message, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 547 super(message, file, line, next); 548 } 549 } 550 551 552 version(Windows) { 553 // FIXME: ugly hack to solve stdin exception problems on Windows: 554 // reading stdin results in StdioException (Bad file descriptor) 555 // this is probably due to http://d.puremagic.com/issues/show_bug.cgi?id=3425 556 private struct stdin { 557 struct ByChunk { // Replicates std.stdio.ByChunk 558 private: 559 ubyte[] chunk_; 560 public: 561 this(size_t size) 562 in { 563 assert(size, "size must be larger than 0"); 564 } 565 do { 566 chunk_ = new ubyte[](size); 567 popFront(); 568 } 569 570 @property bool empty() const { 571 return !std.stdio.stdin.isOpen || std.stdio.stdin.eof; // Ugly, but seems to do the job 572 } 573 @property nothrow ubyte[] front() { return chunk_; } 574 void popFront() { 575 enforce(!empty, "Cannot call popFront on empty range"); 576 chunk_ = stdin.rawRead(chunk_); 577 } 578 } 579 580 import core.sys.windows.windows; 581 static: 582 583 T[] rawRead(T)(T[] buf) { 584 uint bytesRead; 585 auto result = ReadFile(GetStdHandle(STD_INPUT_HANDLE), buf.ptr, cast(int) (buf.length * T.sizeof), &bytesRead, null); 586 587 if (!result) { 588 auto err = GetLastError(); 589 if (err == 38/*ERROR_HANDLE_EOF*/ || err == 109/*ERROR_BROKEN_PIPE*/) // 'good' errors meaning end of input 590 return buf[0..0]; 591 // Some other error, throw it 592 593 char* buffer; 594 scope(exit) LocalFree(buffer); 595 596 // FORMAT_MESSAGE_ALLOCATE_BUFFER = 0x00000100 597 // FORMAT_MESSAGE_FROM_SYSTEM = 0x00001000 598 FormatMessageA(0x1100, null, err, 0, cast(char*)&buffer, 256, null); 599 throw new Exception(to!string(buffer)); 600 } 601 enforce(!(bytesRead % T.sizeof), "I/O error"); 602 return buf[0..bytesRead / T.sizeof]; 603 } 604 605 auto byChunk(size_t sz) { return ByChunk(sz); } 606 607 void close() { 608 std.stdio.stdin.close; 609 } 610 } 611 } 612 613 /// The main interface with the web request 614 class Cgi { 615 public: 616 /// the methods a request can be 617 enum RequestMethod { GET, HEAD, POST, PUT, DELETE, // GET and POST are the ones that really work 618 // these are defined in the standard, but idk if they are useful for anything 619 OPTIONS, TRACE, CONNECT, 620 // These seem new, I have only recently seen them 621 PATCH, MERGE, 622 // this is an extension for when the method is not specified and you want to assume 623 CommandLine } 624 625 626 /+ 627 /++ 628 Cgi provides a per-request memory pool 629 630 +/ 631 void[] allocateMemory(size_t nBytes) { 632 633 } 634 635 /// ditto 636 void[] reallocateMemory(void[] old, size_t nBytes) { 637 638 } 639 640 /// ditto 641 void freeMemory(void[] memory) { 642 643 } 644 +/ 645 646 647 /* 648 import core.runtime; 649 auto args = Runtime.args(); 650 651 we can call the app a few ways: 652 653 1) set up the environment variables and call the app (manually simulating CGI) 654 2) simulate a call automatically: 655 ./app method 'uri' 656 657 for example: 658 ./app get /path?arg arg2=something 659 660 Anything on the uri is treated as query string etc 661 662 on get method, further args are appended to the query string (encoded automatically) 663 on post method, further args are done as post 664 665 666 @name means import from file "name". if name == -, it uses stdin 667 (so info=@- means set info to the value of stdin) 668 669 670 Other arguments include: 671 --cookie name=value (these are all concated together) 672 --header 'X-Something: cool' 673 --referrer 'something' 674 --port 80 675 --remote-address some.ip.address.here 676 --https yes 677 --user-agent 'something' 678 --userpass 'user:pass' 679 --authorization 'Basic base64encoded_user:pass' 680 --accept 'content' // FIXME: better example 681 --last-event-id 'something' 682 --host 'something.com' 683 684 Non-simulation arguments: 685 --port xxx listening port for non-cgi things (valid for the cgi interfaces) 686 --listening-host the ip address the application should listen on, or if you want to use unix domain sockets, it is here you can set them: `--listening-host unix:filename` or, on Linux, `--listening-host abstract:name`. 687 688 */ 689 690 /** Initializes it with command line arguments (for easy testing) */ 691 this(string[] args, void delegate(const(ubyte)[]) _rawDataOutput = null) { 692 rawDataOutput = _rawDataOutput; 693 // these are all set locally so the loop works 694 // without triggering errors in dmd 2.064 695 // we go ahead and set them at the end of it to the this version 696 int port; 697 string referrer; 698 string remoteAddress; 699 string userAgent; 700 string authorization; 701 string origin; 702 string accept; 703 string lastEventId; 704 bool https; 705 string host; 706 RequestMethod requestMethod; 707 string requestUri; 708 string pathInfo; 709 string queryString; 710 711 bool lookingForMethod; 712 bool lookingForUri; 713 string nextArgIs; 714 715 string _cookie; 716 string _queryString; 717 string[][string] _post; 718 string[string] _headers; 719 720 string[] breakUp(string s) { 721 string k, v; 722 auto idx = s.indexOf("="); 723 if(idx == -1) { 724 k = s; 725 } else { 726 k = s[0 .. idx]; 727 v = s[idx + 1 .. $]; 728 } 729 730 return [k, v]; 731 } 732 733 lookingForMethod = true; 734 735 scriptName = args[0]; 736 scriptFileName = args[0]; 737 738 environmentVariables = cast(const) environment.toAA; 739 740 foreach(arg; args[1 .. $]) { 741 if(arg.startsWith("--")) { 742 nextArgIs = arg[2 .. $]; 743 } else if(nextArgIs.length) { 744 if (nextArgIs == "cookie") { 745 auto info = breakUp(arg); 746 if(_cookie.length) 747 _cookie ~= "; "; 748 _cookie ~= std.uri.encodeComponent(info[0]) ~ "=" ~ std.uri.encodeComponent(info[1]); 749 } 750 else if (nextArgIs == "port") { 751 port = to!int(arg); 752 } 753 else if (nextArgIs == "referrer") { 754 referrer = arg; 755 } 756 else if (nextArgIs == "remote-address") { 757 remoteAddress = arg; 758 } 759 else if (nextArgIs == "user-agent") { 760 userAgent = arg; 761 } 762 else if (nextArgIs == "authorization") { 763 authorization = arg; 764 } 765 else if (nextArgIs == "userpass") { 766 authorization = "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (arg)).idup; 767 } 768 else if (nextArgIs == "origin") { 769 origin = arg; 770 } 771 else if (nextArgIs == "accept") { 772 accept = arg; 773 } 774 else if (nextArgIs == "last-event-id") { 775 lastEventId = arg; 776 } 777 else if (nextArgIs == "https") { 778 if(arg == "yes") 779 https = true; 780 } 781 else if (nextArgIs == "header") { 782 string thing, other; 783 auto idx = arg.indexOf(":"); 784 if(idx == -1) 785 throw new Exception("need a colon in a http header"); 786 thing = arg[0 .. idx]; 787 other = arg[idx + 1.. $]; 788 _headers[thing.strip.toLower()] = other.strip; 789 } 790 else if (nextArgIs == "host") { 791 host = arg; 792 } 793 // else 794 // skip, we don't know it but that's ok, it might be used elsewhere so no error 795 796 nextArgIs = null; 797 } else if(lookingForMethod) { 798 lookingForMethod = false; 799 lookingForUri = true; 800 801 if(arg.asLowerCase().equal("commandline")) 802 requestMethod = RequestMethod.CommandLine; 803 else 804 requestMethod = to!RequestMethod(arg.toUpper()); 805 } else if(lookingForUri) { 806 lookingForUri = false; 807 808 requestUri = arg; 809 810 auto idx = arg.indexOf("?"); 811 if(idx == -1) 812 pathInfo = arg; 813 else { 814 pathInfo = arg[0 .. idx]; 815 _queryString = arg[idx + 1 .. $]; 816 } 817 } else { 818 // it is an argument of some sort 819 if(requestMethod == Cgi.RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 820 auto parts = breakUp(arg); 821 _post[parts[0]] ~= parts[1]; 822 allPostNamesInOrder ~= parts[0]; 823 allPostValuesInOrder ~= parts[1]; 824 } else { 825 if(_queryString.length) 826 _queryString ~= "&"; 827 auto parts = breakUp(arg); 828 _queryString ~= std.uri.encodeComponent(parts[0]) ~ "=" ~ std.uri.encodeComponent(parts[1]); 829 } 830 } 831 } 832 833 acceptsGzip = false; 834 keepAliveRequested = false; 835 requestHeaders = cast(immutable) _headers; 836 837 cookie = _cookie; 838 cookiesArray = getCookieArray(); 839 cookies = keepLastOf(cookiesArray); 840 841 queryString = _queryString; 842 getArray = cast(immutable) decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 843 get = keepLastOf(getArray); 844 845 postArray = cast(immutable) _post; 846 post = keepLastOf(_post); 847 848 // FIXME 849 filesArray = null; 850 files = null; 851 852 isCalledWithCommandLineArguments = true; 853 854 this.port = port; 855 this.referrer = referrer; 856 this.remoteAddress = remoteAddress; 857 this.userAgent = userAgent; 858 this.authorization = authorization; 859 this.origin = origin; 860 this.accept = accept; 861 this.lastEventId = lastEventId; 862 this.https = https; 863 this.host = host; 864 this.requestMethod = requestMethod; 865 this.requestUri = requestUri; 866 this.pathInfo = pathInfo; 867 this.queryString = queryString; 868 this.postBody = null; 869 } 870 871 private { 872 string[] allPostNamesInOrder; 873 string[] allPostValuesInOrder; 874 string[] allGetNamesInOrder; 875 string[] allGetValuesInOrder; 876 } 877 878 CgiConnectionHandle getOutputFileHandle() { 879 return _outputFileHandle; 880 } 881 882 CgiConnectionHandle _outputFileHandle = INVALID_CGI_CONNECTION_HANDLE; 883 884 /** Initializes it using a CGI or CGI-like interface */ 885 this(long maxContentLength = defaultMaxContentLength, 886 // use this to override the environment variable listing 887 in string[string] env = null, 888 // and this should return a chunk of data. return empty when done 889 const(ubyte)[] delegate() readdata = null, 890 // finally, use this to do custom output if needed 891 void delegate(const(ubyte)[]) _rawDataOutput = null, 892 // to flush teh custom output 893 void delegate() _flush = null 894 ) 895 { 896 897 // these are all set locally so the loop works 898 // without triggering errors in dmd 2.064 899 // we go ahead and set them at the end of it to the this version 900 int port; 901 string referrer; 902 string remoteAddress; 903 string userAgent; 904 string authorization; 905 string origin; 906 string accept; 907 string lastEventId; 908 bool https; 909 string host; 910 RequestMethod requestMethod; 911 string requestUri; 912 string pathInfo; 913 string queryString; 914 915 916 917 isCalledWithCommandLineArguments = false; 918 rawDataOutput = _rawDataOutput; 919 flushDelegate = _flush; 920 auto getenv = delegate string(string var) { 921 if(env is null) 922 return std.process.environment.get(var); 923 auto e = var in env; 924 if(e is null) 925 return null; 926 return *e; 927 }; 928 929 environmentVariables = env is null ? 930 cast(const) environment.toAA : 931 env; 932 933 // fetching all the request headers 934 string[string] requestHeadersHere; 935 foreach(k, v; env is null ? cast(const) environment.toAA() : env) { 936 if(k.startsWith("HTTP_")) { 937 requestHeadersHere[replace(k["HTTP_".length .. $].toLower(), "_", "-")] = v; 938 } 939 } 940 941 this.requestHeaders = assumeUnique(requestHeadersHere); 942 943 requestUri = getenv("REQUEST_URI"); 944 945 cookie = getenv("HTTP_COOKIE"); 946 cookiesArray = getCookieArray(); 947 cookies = keepLastOf(cookiesArray); 948 949 referrer = getenv("HTTP_REFERER"); 950 userAgent = getenv("HTTP_USER_AGENT"); 951 remoteAddress = getenv("REMOTE_ADDR"); 952 host = getenv("HTTP_HOST"); 953 pathInfo = getenv("PATH_INFO"); 954 955 queryString = getenv("QUERY_STRING"); 956 scriptName = getenv("SCRIPT_NAME"); 957 { 958 import core.runtime; 959 auto sfn = getenv("SCRIPT_FILENAME"); 960 scriptFileName = sfn.length ? sfn : (Runtime.args.length ? Runtime.args[0] : null); 961 } 962 963 bool iis = false; 964 965 // Because IIS doesn't pass requestUri, we simulate it here if it's empty. 966 if(requestUri.length == 0) { 967 // IIS sometimes includes the script name as part of the path info - we don't want that 968 if(pathInfo.length >= scriptName.length && (pathInfo[0 .. scriptName.length] == scriptName)) 969 pathInfo = pathInfo[scriptName.length .. $]; 970 971 requestUri = scriptName ~ pathInfo ~ (queryString.length ? ("?" ~ queryString) : ""); 972 973 iis = true; // FIXME HACK - used in byChunk below - see bugzilla 6339 974 975 // FIXME: this works for apache and iis... but what about others? 976 } 977 978 979 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 980 getArray = assumeUnique(ugh); 981 get = keepLastOf(getArray); 982 983 984 // NOTE: on shitpache, you need to specifically forward this 985 authorization = getenv("HTTP_AUTHORIZATION"); 986 // this is a hack because Apache is a shitload of fuck and 987 // refuses to send the real header to us. Compatible 988 // programs should send both the standard and X- versions 989 990 // NOTE: if you have access to .htaccess or httpd.conf, you can make this 991 // unnecessary with mod_rewrite, so it is commented 992 993 //if(authorization.length == 0) // if the std is there, use it 994 // authorization = getenv("HTTP_X_AUTHORIZATION"); 995 996 // the REDIRECT_HTTPS check is here because with an Apache hack, the port can become wrong 997 if(getenv("SERVER_PORT").length && getenv("REDIRECT_HTTPS") != "on") 998 port = to!int(getenv("SERVER_PORT")); 999 else 1000 port = 0; // this was probably called from the command line 1001 1002 auto ae = getenv("HTTP_ACCEPT_ENCODING"); 1003 if(ae.length && ae.indexOf("gzip") != -1) 1004 acceptsGzip = true; 1005 1006 accept = getenv("HTTP_ACCEPT"); 1007 lastEventId = getenv("HTTP_LAST_EVENT_ID"); 1008 1009 auto ka = getenv("HTTP_CONNECTION"); 1010 if(ka.length && ka.asLowerCase().canFind("keep-alive")) 1011 keepAliveRequested = true; 1012 1013 auto or = getenv("HTTP_ORIGIN"); 1014 origin = or; 1015 1016 auto rm = getenv("REQUEST_METHOD"); 1017 if(rm.length) 1018 requestMethod = to!RequestMethod(getenv("REQUEST_METHOD")); 1019 else 1020 requestMethod = RequestMethod.CommandLine; 1021 1022 // FIXME: hack on REDIRECT_HTTPS; this is there because the work app uses mod_rewrite which loses the https flag! So I set it with [E=HTTPS=%HTTPS] or whatever but then it gets translated to here so i want it to still work. This is arguably wrong but meh. 1023 https = (getenv("HTTPS") == "on" || getenv("REDIRECT_HTTPS") == "on"); 1024 1025 // FIXME: DOCUMENT_ROOT? 1026 1027 // FIXME: what about PUT? 1028 if(requestMethod == RequestMethod.POST || requestMethod == Cgi.RequestMethod.PATCH || requestMethod == Cgi.RequestMethod.PUT || requestMethod == Cgi.RequestMethod.CommandLine) { 1029 version(preserveData) // a hack to make forwarding simpler 1030 immutable(ubyte)[] data; 1031 size_t amountReceived = 0; 1032 auto contentType = getenv("CONTENT_TYPE"); 1033 1034 // FIXME: is this ever not going to be set? I guess it depends 1035 // on if the server de-chunks and buffers... seems like it has potential 1036 // to be slow if they did that. The spec says it is always there though. 1037 // And it has worked reliably for me all year in the live environment, 1038 // but some servers might be different. 1039 auto cls = getenv("CONTENT_LENGTH"); 1040 auto contentLength = to!size_t(cls.length ? cls : "0"); 1041 1042 immutable originalContentLength = contentLength; 1043 if(contentLength) { 1044 if(maxContentLength > 0 && contentLength > maxContentLength) { 1045 setResponseStatus("413 Request entity too large"); 1046 write("You tried to upload a file that is too large."); 1047 close(); 1048 throw new Exception("POST too large"); 1049 } 1050 prepareForIncomingDataChunks(contentType, contentLength); 1051 1052 1053 int processChunk(in ubyte[] chunk) { 1054 if(chunk.length > contentLength) { 1055 handleIncomingDataChunk(chunk[0..contentLength]); 1056 amountReceived += contentLength; 1057 contentLength = 0; 1058 return 1; 1059 } else { 1060 handleIncomingDataChunk(chunk); 1061 contentLength -= chunk.length; 1062 amountReceived += chunk.length; 1063 } 1064 if(contentLength == 0) 1065 return 1; 1066 1067 onRequestBodyDataReceived(amountReceived, originalContentLength); 1068 return 0; 1069 } 1070 1071 1072 if(readdata is null) { 1073 foreach(ubyte[] chunk; stdin.byChunk(iis ? contentLength : 4096)) 1074 if(processChunk(chunk)) 1075 break; 1076 } else { 1077 // we have a custom data source.. 1078 auto chunk = readdata(); 1079 while(chunk.length) { 1080 if(processChunk(chunk)) 1081 break; 1082 chunk = readdata(); 1083 } 1084 } 1085 1086 onRequestBodyDataReceived(amountReceived, originalContentLength); 1087 postArray = assumeUnique(pps._post); 1088 filesArray = assumeUnique(pps._files); 1089 files = keepLastOf(filesArray); 1090 post = keepLastOf(postArray); 1091 this.postBody = pps.postBody; 1092 cleanUpPostDataState(); 1093 } 1094 1095 version(preserveData) 1096 originalPostData = data; 1097 } 1098 // fixme: remote_user script name 1099 1100 1101 this.port = port; 1102 this.referrer = referrer; 1103 this.remoteAddress = remoteAddress; 1104 this.userAgent = userAgent; 1105 this.authorization = authorization; 1106 this.origin = origin; 1107 this.accept = accept; 1108 this.lastEventId = lastEventId; 1109 this.https = https; 1110 this.host = host; 1111 this.requestMethod = requestMethod; 1112 this.requestUri = requestUri; 1113 this.pathInfo = pathInfo; 1114 this.queryString = queryString; 1115 } 1116 1117 /// Cleans up any temporary files. Do not use the object 1118 /// after calling this. 1119 /// 1120 /// NOTE: it is called automatically by GenericMain 1121 // FIXME: this should be called if the constructor fails too, if it has created some garbage... 1122 void dispose() { 1123 foreach(file; files) { 1124 if(!file.contentInMemory) 1125 if(std.file.exists(file.contentFilename)) 1126 std.file.remove(file.contentFilename); 1127 } 1128 } 1129 1130 private { 1131 struct PostParserState { 1132 string contentType; 1133 string boundary; 1134 string localBoundary; // the ones used at the end or something lol 1135 bool isMultipart; 1136 bool needsSavedBody; 1137 1138 ulong expectedLength; 1139 ulong contentConsumed; 1140 immutable(ubyte)[] buffer; 1141 1142 // multipart parsing state 1143 int whatDoWeWant; 1144 bool weHaveAPart; 1145 string[] thisOnesHeaders; 1146 immutable(ubyte)[] thisOnesData; 1147 1148 string postBody; 1149 1150 UploadedFile piece; 1151 bool isFile = false; 1152 1153 size_t memoryCommitted; 1154 1155 // do NOT keep mutable references to these anywhere! 1156 // I assume they are unique in the constructor once we're all done getting data. 1157 string[][string] _post; 1158 UploadedFile[][string] _files; 1159 } 1160 1161 PostParserState pps; 1162 } 1163 1164 /// This represents a file the user uploaded via a POST request. 1165 static struct UploadedFile { 1166 /// If you want to create one of these structs for yourself from some data, 1167 /// use this function. 1168 static UploadedFile fromData(immutable(void)[] data, string name = null) { 1169 Cgi.UploadedFile f; 1170 f.filename = name; 1171 f.content = cast(immutable(ubyte)[]) data; 1172 f.contentInMemory = true; 1173 return f; 1174 } 1175 1176 string name; /// The name of the form element. 1177 string filename; /// The filename the user set. 1178 string contentType; /// The MIME type the user's browser reported. (Not reliable.) 1179 1180 /** 1181 For small files, cgi.d will buffer the uploaded file in memory, and make it 1182 directly accessible to you through the content member. I find this very convenient 1183 and somewhat efficient, since it can avoid hitting the disk entirely. (I 1184 often want to inspect and modify the file anyway!) 1185 1186 I find the file is very large, it is undesirable to eat that much memory just 1187 for a file buffer. In those cases, if you pass a large enough value for maxContentLength 1188 to the constructor so they are accepted, cgi.d will write the content to a temporary 1189 file that you can re-read later. 1190 1191 You can override this behavior by subclassing Cgi and overriding the protected 1192 handlePostChunk method. Note that the object is not initialized when you 1193 write that method - the http headers are available, but the cgi.post method 1194 is not. You may parse the file as it streams in using this method. 1195 1196 1197 Anyway, if the file is small enough to be in memory, contentInMemory will be 1198 set to true, and the content is available in the content member. 1199 1200 If not, contentInMemory will be set to false, and the content saved in a file, 1201 whose name will be available in the contentFilename member. 1202 1203 1204 Tip: if you know you are always dealing with small files, and want the convenience 1205 of ignoring this member, construct Cgi with a small maxContentLength. Then, if 1206 a large file comes in, it simply throws an exception (and HTTP error response) 1207 instead of trying to handle it. 1208 1209 The default value of maxContentLength in the constructor is for small files. 1210 */ 1211 bool contentInMemory = true; // the default ought to always be true 1212 immutable(ubyte)[] content; /// The actual content of the file, if contentInMemory == true 1213 string contentFilename; /// the file where we dumped the content, if contentInMemory == false. Note that if you want to keep it, you MUST move the file, since otherwise it is considered garbage when cgi is disposed. 1214 1215 /// 1216 ulong fileSize() { 1217 if(contentInMemory) 1218 return content.length; 1219 import std.file; 1220 return std.file.getSize(contentFilename); 1221 1222 } 1223 1224 /// 1225 void writeToFile(string filenameToSaveTo) const { 1226 import std.file; 1227 if(contentInMemory) 1228 std.file.write(filenameToSaveTo, content); 1229 else 1230 std.file.rename(contentFilename, filenameToSaveTo); 1231 } 1232 } 1233 1234 // given a content type and length, decide what we're going to do with the data.. 1235 protected void prepareForIncomingDataChunks(string contentType, ulong contentLength) { 1236 pps.expectedLength = contentLength; 1237 1238 auto terminator = contentType.indexOf(";"); 1239 if(terminator == -1) 1240 terminator = contentType.length; 1241 1242 pps.contentType = contentType[0 .. terminator]; 1243 auto b = contentType[terminator .. $]; 1244 if(b.length) { 1245 auto idx = b.indexOf("boundary="); 1246 if(idx != -1) { 1247 pps.boundary = b[idx + "boundary=".length .. $]; 1248 pps.localBoundary = "\r\n--" ~ pps.boundary; 1249 } 1250 } 1251 1252 // while a content type SHOULD be sent according to the RFC, it is 1253 // not required. We're told we SHOULD guess by looking at the content 1254 // but it seems to me that this only happens when it is urlencoded. 1255 if(pps.contentType == "application/x-www-form-urlencoded" || pps.contentType == "") { 1256 pps.isMultipart = false; 1257 pps.needsSavedBody = false; 1258 } else if(pps.contentType == "multipart/form-data") { 1259 pps.isMultipart = true; 1260 enforce(pps.boundary.length, "no boundary"); 1261 } else if(pps.contentType == "text/xml") { // FIXME: could this be special and load the post params 1262 // save the body so the application can handle it 1263 pps.isMultipart = false; 1264 pps.needsSavedBody = true; 1265 } else if(pps.contentType == "application/json") { // FIXME: this could prolly try to load post params too 1266 // save the body so the application can handle it 1267 pps.needsSavedBody = true; 1268 pps.isMultipart = false; 1269 } else { 1270 // the rest is 100% handled by the application. just save the body and send it to them 1271 pps.needsSavedBody = true; 1272 pps.isMultipart = false; 1273 } 1274 } 1275 1276 // handles streaming POST data. If you handle some other content type, you should 1277 // override this. If the data isn't the content type you want, you ought to call 1278 // super.handleIncomingDataChunk so regular forms and files still work. 1279 1280 // FIXME: I do some copying in here that I'm pretty sure is unnecessary, and the 1281 // file stuff I'm sure is inefficient. But, my guess is the real bottleneck is network 1282 // input anyway, so I'm not going to get too worked up about it right now. 1283 protected void handleIncomingDataChunk(const(ubyte)[] chunk) { 1284 if(chunk.length == 0) 1285 return; 1286 assert(chunk.length <= 32 * 1024 * 1024); // we use chunk size as a memory constraint thing, so 1287 // if we're passed big chunks, it might throw unnecessarily. 1288 // just pass it smaller chunks at a time. 1289 if(pps.isMultipart) { 1290 // multipart/form-data 1291 1292 1293 // FIXME: this might want to be factored out and factorized 1294 // need to make sure the stream hooks actually work. 1295 void pieceHasNewContent() { 1296 // we just grew the piece's buffer. Do we have to switch to file backing? 1297 if(pps.piece.contentInMemory) { 1298 if(pps.piece.content.length <= 10 * 1024 * 1024) 1299 // meh, I'm ok with it. 1300 return; 1301 else { 1302 // this is too big. 1303 if(!pps.isFile) 1304 throw new Exception("Request entity too large"); // a variable this big is kinda ridiculous, just reject it. 1305 else { 1306 // a file this large is probably acceptable though... let's use a backing file. 1307 pps.piece.contentInMemory = false; 1308 // FIXME: say... how do we intend to delete these things? cgi.dispose perhaps. 1309 1310 int count = 0; 1311 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1312 // odds are this loop will never be entered, but we want it just in case. 1313 while(std.file.exists(pps.piece.contentFilename)) { 1314 count++; 1315 pps.piece.contentFilename = getTempDirectory() ~ "arsd_cgi_uploaded_file_" ~ to!string(getUtcTime()) ~ "-" ~ to!string(count); 1316 } 1317 // I hope this creates the file pretty quickly, or the loop might be useless... 1318 // FIXME: maybe I should write some kind of custom transaction here. 1319 std.file.write(pps.piece.contentFilename, pps.piece.content); 1320 1321 pps.piece.content = null; 1322 } 1323 } 1324 } else { 1325 // it's already in a file, so just append it to what we have 1326 if(pps.piece.content.length) { 1327 // FIXME: this is surely very inefficient... we'll be calling this by 4kb chunk... 1328 std.file.append(pps.piece.contentFilename, pps.piece.content); 1329 pps.piece.content = null; 1330 } 1331 } 1332 } 1333 1334 1335 void commitPart() { 1336 if(!pps.weHaveAPart) 1337 return; 1338 1339 pieceHasNewContent(); // be sure the new content is handled every time 1340 1341 if(pps.isFile) { 1342 // I'm not sure if other environments put files in post or not... 1343 // I used to not do it, but I think I should, since it is there... 1344 pps._post[pps.piece.name] ~= pps.piece.filename; 1345 pps._files[pps.piece.name] ~= pps.piece; 1346 1347 allPostNamesInOrder ~= pps.piece.name; 1348 allPostValuesInOrder ~= pps.piece.filename; 1349 } else { 1350 pps._post[pps.piece.name] ~= cast(string) pps.piece.content; 1351 1352 allPostNamesInOrder ~= pps.piece.name; 1353 allPostValuesInOrder ~= cast(string) pps.piece.content; 1354 } 1355 1356 /* 1357 stderr.writeln("RECEIVED: ", pps.piece.name, "=", 1358 pps.piece.content.length < 1000 1359 ? 1360 to!string(pps.piece.content) 1361 : 1362 "too long"); 1363 */ 1364 1365 // FIXME: the limit here 1366 pps.memoryCommitted += pps.piece.content.length; 1367 1368 pps.weHaveAPart = false; 1369 pps.whatDoWeWant = 1; 1370 pps.thisOnesHeaders = null; 1371 pps.thisOnesData = null; 1372 1373 pps.piece = UploadedFile.init; 1374 pps.isFile = false; 1375 } 1376 1377 void acceptChunk() { 1378 pps.buffer ~= chunk; 1379 chunk = null; // we've consumed it into the buffer, so keeping it just brings confusion 1380 } 1381 1382 immutable(ubyte)[] consume(size_t howMuch) { 1383 pps.contentConsumed += howMuch; 1384 auto ret = pps.buffer[0 .. howMuch]; 1385 pps.buffer = pps.buffer[howMuch .. $]; 1386 return ret; 1387 } 1388 1389 dataConsumptionLoop: do { 1390 switch(pps.whatDoWeWant) { 1391 default: assert(0); 1392 case 0: 1393 acceptChunk(); 1394 // the format begins with two extra leading dashes, then we should be at the boundary 1395 if(pps.buffer.length < 2) 1396 return; 1397 assert(pps.buffer[0] == '-', "no leading dash"); 1398 consume(1); 1399 assert(pps.buffer[0] == '-', "no second leading dash"); 1400 consume(1); 1401 1402 pps.whatDoWeWant = 1; 1403 goto case 1; 1404 /* fallthrough */ 1405 case 1: // looking for headers 1406 // here, we should be lined up right at the boundary, which is followed by a \r\n 1407 1408 // want to keep the buffer under control in case we're under attack 1409 //stderr.writeln("here once"); 1410 //if(pps.buffer.length + chunk.length > 70 * 1024) // they should be < 1 kb really.... 1411 // throw new Exception("wtf is up with the huge mime part headers"); 1412 1413 acceptChunk(); 1414 1415 if(pps.buffer.length < pps.boundary.length) 1416 return; // not enough data, since there should always be a boundary here at least 1417 1418 if(pps.contentConsumed + pps.boundary.length + 6 == pps.expectedLength) { 1419 assert(pps.buffer.length == pps.boundary.length + 4 + 2); // --, --, and \r\n 1420 // we *should* be at the end here! 1421 assert(pps.buffer[0] == '-'); 1422 consume(1); 1423 assert(pps.buffer[0] == '-'); 1424 consume(1); 1425 1426 // the message is terminated by --BOUNDARY--\r\n (after a \r\n leading to the boundary) 1427 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1428 "not lined up on boundary " ~ pps.boundary); 1429 consume(pps.boundary.length); 1430 1431 assert(pps.buffer[0] == '-'); 1432 consume(1); 1433 assert(pps.buffer[0] == '-'); 1434 consume(1); 1435 1436 assert(pps.buffer[0] == '\r'); 1437 consume(1); 1438 assert(pps.buffer[0] == '\n'); 1439 consume(1); 1440 1441 assert(pps.buffer.length == 0); 1442 assert(pps.contentConsumed == pps.expectedLength); 1443 break dataConsumptionLoop; // we're done! 1444 } else { 1445 // we're not done yet. We should be lined up on a boundary. 1446 1447 // But, we want to ensure the headers are here before we consume anything! 1448 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1449 if(headerEndLocation == -1) 1450 return; // they *should* all be here, so we can handle them all at once. 1451 1452 assert(pps.buffer[0 .. pps.boundary.length] == cast(const(ubyte[])) pps.boundary, 1453 "not lined up on boundary " ~ pps.boundary); 1454 1455 consume(pps.boundary.length); 1456 // the boundary is always followed by a \r\n 1457 assert(pps.buffer[0] == '\r'); 1458 consume(1); 1459 assert(pps.buffer[0] == '\n'); 1460 consume(1); 1461 } 1462 1463 // re-running since by consuming the boundary, we invalidate the old index. 1464 auto headerEndLocation = locationOf(pps.buffer, "\r\n\r\n"); 1465 assert(headerEndLocation >= 0, "no header"); 1466 auto thisOnesHeaders = pps.buffer[0..headerEndLocation]; 1467 1468 consume(headerEndLocation + 4); // The +4 is the \r\n\r\n that caps it off 1469 1470 pps.thisOnesHeaders = split(cast(string) thisOnesHeaders, "\r\n"); 1471 1472 // now we'll parse the headers 1473 foreach(h; pps.thisOnesHeaders) { 1474 auto p = h.indexOf(":"); 1475 assert(p != -1, "no colon in header, got " ~ to!string(pps.thisOnesHeaders)); 1476 string hn = h[0..p]; 1477 string hv = h[p+2..$]; 1478 1479 switch(hn.toLower) { 1480 default: assert(0); 1481 case "content-disposition": 1482 auto info = hv.split("; "); 1483 foreach(i; info[1..$]) { // skipping the form-data 1484 auto o = i.split("="); // FIXME 1485 string pn = o[0]; 1486 string pv = o[1][1..$-1]; 1487 1488 if(pn == "name") { 1489 pps.piece.name = pv; 1490 } else if (pn == "filename") { 1491 pps.piece.filename = pv; 1492 pps.isFile = true; 1493 } 1494 } 1495 break; 1496 case "content-type": 1497 pps.piece.contentType = hv; 1498 break; 1499 } 1500 } 1501 1502 pps.whatDoWeWant++; // move to the next step - the data 1503 break; 1504 case 2: 1505 // when we get here, pps.buffer should contain our first chunk of data 1506 1507 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // we might buffer quite a bit but not much 1508 throw new Exception("wtf is up with the huge mime part buffer"); 1509 1510 acceptChunk(); 1511 1512 // so the trick is, we want to process all the data up to the boundary, 1513 // but what if the chunk's end cuts the boundary off? If we're unsure, we 1514 // want to wait for the next chunk. We start by looking for the whole boundary 1515 // in the buffer somewhere. 1516 1517 auto boundaryLocation = locationOf(pps.buffer, pps.localBoundary); 1518 // assert(boundaryLocation != -1, "should have seen "~to!string(cast(ubyte[]) pps.localBoundary)~" in " ~ to!string(pps.buffer)); 1519 if(boundaryLocation != -1) { 1520 // this is easy - we can see it in it's entirety! 1521 1522 pps.piece.content ~= consume(boundaryLocation); 1523 1524 assert(pps.buffer[0] == '\r'); 1525 consume(1); 1526 assert(pps.buffer[0] == '\n'); 1527 consume(1); 1528 assert(pps.buffer[0] == '-'); 1529 consume(1); 1530 assert(pps.buffer[0] == '-'); 1531 consume(1); 1532 // the boundary here is always preceded by \r\n--, which is why we used localBoundary instead of boundary to locate it. Cut that off. 1533 pps.weHaveAPart = true; 1534 pps.whatDoWeWant = 1; // back to getting headers for the next part 1535 1536 commitPart(); // we're done here 1537 } else { 1538 // we can't see the whole thing, but what if there's a partial boundary? 1539 1540 enforce(pps.localBoundary.length < 128); // the boundary ought to be less than a line... 1541 assert(pps.localBoundary.length > 1); // should already be sane but just in case 1542 bool potentialBoundaryFound = false; 1543 1544 boundaryCheck: for(int a = 1; a < pps.localBoundary.length; a++) { 1545 // we grow the boundary a bit each time. If we think it looks the 1546 // same, better pull another chunk to be sure it's not the end. 1547 // Starting small because exiting the loop early is desirable, since 1548 // we're not keeping any ambiguity and 1 / 256 chance of exiting is 1549 // the best we can do. 1550 if(a > pps.buffer.length) 1551 break; // FIXME: is this right? 1552 assert(a <= pps.buffer.length); 1553 assert(a > 0); 1554 if(std.algorithm.endsWith(pps.buffer, pps.localBoundary[0 .. a])) { 1555 // ok, there *might* be a boundary here, so let's 1556 // not treat the end as data yet. The rest is good to 1557 // use though, since if there was a boundary there, we'd 1558 // have handled it up above after locationOf. 1559 1560 pps.piece.content ~= pps.buffer[0 .. $ - a]; 1561 consume(pps.buffer.length - a); 1562 pieceHasNewContent(); 1563 potentialBoundaryFound = true; 1564 break boundaryCheck; 1565 } 1566 } 1567 1568 if(!potentialBoundaryFound) { 1569 // we can consume the whole thing 1570 pps.piece.content ~= pps.buffer; 1571 pieceHasNewContent(); 1572 consume(pps.buffer.length); 1573 } else { 1574 // we found a possible boundary, but there was 1575 // insufficient data to be sure. 1576 assert(pps.buffer == cast(const(ubyte[])) pps.localBoundary[0 .. pps.buffer.length]); 1577 1578 return; // wait for the next chunk. 1579 } 1580 } 1581 } 1582 } while(pps.buffer.length); 1583 1584 // btw all boundaries except the first should have a \r\n before them 1585 } else { 1586 // application/x-www-form-urlencoded and application/json 1587 1588 // not using maxContentLength because that might be cranked up to allow 1589 // large file uploads. We can handle them, but a huge post[] isn't any good. 1590 if(pps.buffer.length + chunk.length > 8 * 1024 * 1024) // surely this is plenty big enough 1591 throw new Exception("wtf is up with such a gigantic form submission????"); 1592 1593 pps.buffer ~= chunk; 1594 1595 // simple handling, but it works... until someone bombs us with gigabytes of crap at least... 1596 if(pps.buffer.length == pps.expectedLength) { 1597 if(pps.needsSavedBody) 1598 pps.postBody = cast(string) pps.buffer; 1599 else 1600 pps._post = decodeVariables(cast(string) pps.buffer, "&", &allPostNamesInOrder, &allPostValuesInOrder); 1601 version(preserveData) 1602 originalPostData = pps.buffer; 1603 } else { 1604 // just for debugging 1605 } 1606 } 1607 } 1608 1609 protected void cleanUpPostDataState() { 1610 pps = PostParserState.init; 1611 } 1612 1613 /// you can override this function to somehow react 1614 /// to an upload in progress. 1615 /// 1616 /// Take note that parts of the CGI object is not yet 1617 /// initialized! Stuff from HTTP headers, including get[], is usable. 1618 /// But, none of post[] is usable, and you cannot write here. That's 1619 /// why this method is const - mutating the object won't do much anyway. 1620 /// 1621 /// My idea here was so you can output a progress bar or 1622 /// something to a cooperative client (see arsd.rtud for a potential helper) 1623 /// 1624 /// The default is to do nothing. Subclass cgi and use the 1625 /// CustomCgiMain mixin to do something here. 1626 void onRequestBodyDataReceived(size_t receivedSoFar, size_t totalExpected) const { 1627 // This space intentionally left blank. 1628 } 1629 1630 /// Initializes the cgi from completely raw HTTP data. The ir must have a Socket source. 1631 /// *closeConnection will be set to true if you should close the connection after handling this request 1632 this(BufferedInputRange ir, bool* closeConnection) { 1633 isCalledWithCommandLineArguments = false; 1634 import al = std.algorithm; 1635 1636 immutable(ubyte)[] data; 1637 1638 void rdo(const(ubyte)[] d) { 1639 //import std.stdio; writeln(d); 1640 sendAll(ir.source, d); 1641 } 1642 1643 auto ira = ir.source.remoteAddress(); 1644 auto irLocalAddress = ir.source.localAddress(); 1645 1646 ushort port = 80; 1647 if(auto ia = cast(InternetAddress) irLocalAddress) { 1648 port = ia.port; 1649 } else if(auto ia = cast(Internet6Address) irLocalAddress) { 1650 port = ia.port; 1651 } 1652 1653 // that check for UnixAddress is to work around a Phobos bug 1654 // see: https://github.com/dlang/phobos/pull/7383 1655 // but this might be more useful anyway tbh for this case 1656 version(Posix) 1657 this(ir, ira is null ? null : cast(UnixAddress) ira ? "unix:" : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1658 else 1659 this(ir, ira is null ? null : ira.toString(), port, 0, false, &rdo, null, closeConnection); 1660 } 1661 1662 /** 1663 Initializes it from raw HTTP request data. GenericMain uses this when you compile with -version=embedded_httpd. 1664 1665 NOTE: If you are behind a reverse proxy, the values here might not be what you expect.... it will use X-Forwarded-For for remote IP and X-Forwarded-Host for host 1666 1667 Params: 1668 inputData = the incoming data, including headers and other raw http data. 1669 When the constructor exits, it will leave this range exactly at the start of 1670 the next request on the connection (if there is one). 1671 1672 address = the IP address of the remote user 1673 _port = the port number of the connection 1674 pathInfoStarts = the offset into the path component of the http header where the SCRIPT_NAME ends and the PATH_INFO begins. 1675 _https = if this connection is encrypted (note that the input data must not actually be encrypted) 1676 _rawDataOutput = delegate to accept response data. It should write to the socket or whatever; Cgi does all the needed processing to speak http. 1677 _flush = if _rawDataOutput buffers, this delegate should flush the buffer down the wire 1678 closeConnection = if the request asks to close the connection, *closeConnection == true. 1679 */ 1680 this( 1681 BufferedInputRange inputData, 1682 // string[] headers, immutable(ubyte)[] data, 1683 string address, ushort _port, 1684 int pathInfoStarts = 0, // use this if you know the script name, like if this is in a folder in a bigger web environment 1685 bool _https = false, 1686 void delegate(const(ubyte)[]) _rawDataOutput = null, 1687 void delegate() _flush = null, 1688 // this pointer tells if the connection is supposed to be closed after we handle this 1689 bool* closeConnection = null) 1690 { 1691 // these are all set locally so the loop works 1692 // without triggering errors in dmd 2.064 1693 // we go ahead and set them at the end of it to the this version 1694 int port; 1695 string referrer; 1696 string remoteAddress; 1697 string userAgent; 1698 string authorization; 1699 string origin; 1700 string accept; 1701 string lastEventId; 1702 bool https; 1703 string host; 1704 RequestMethod requestMethod; 1705 string requestUri; 1706 string pathInfo; 1707 string queryString; 1708 string scriptName; 1709 string[string] get; 1710 string[][string] getArray; 1711 bool keepAliveRequested; 1712 bool acceptsGzip; 1713 string cookie; 1714 1715 1716 1717 environmentVariables = cast(const) environment.toAA; 1718 1719 idlol = inputData; 1720 1721 isCalledWithCommandLineArguments = false; 1722 1723 https = _https; 1724 port = _port; 1725 1726 rawDataOutput = _rawDataOutput; 1727 flushDelegate = _flush; 1728 nph = true; 1729 1730 remoteAddress = address; 1731 1732 // streaming parser 1733 import al = std.algorithm; 1734 1735 // FIXME: tis cast is technically wrong, but Phobos deprecated al.indexOf... for some reason. 1736 auto idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1737 while(idx == -1) { 1738 inputData.popFront(0); 1739 idx = indexOf(cast(string) inputData.front(), "\r\n\r\n"); 1740 } 1741 1742 assert(idx != -1); 1743 1744 1745 string contentType = ""; 1746 string[string] requestHeadersHere; 1747 1748 size_t contentLength; 1749 1750 bool isChunked; 1751 1752 { 1753 import core.runtime; 1754 scriptFileName = Runtime.args.length ? Runtime.args[0] : null; 1755 } 1756 1757 1758 int headerNumber = 0; 1759 foreach(line; al.splitter(inputData.front()[0 .. idx], "\r\n")) 1760 if(line.length) { 1761 headerNumber++; 1762 auto header = cast(string) line.idup; 1763 if(headerNumber == 1) { 1764 // request line 1765 auto parts = al.splitter(header, " "); 1766 requestMethod = to!RequestMethod(parts.front); 1767 parts.popFront(); 1768 requestUri = parts.front; 1769 1770 // FIXME: the requestUri could be an absolute path!!! should I rename it or something? 1771 scriptName = requestUri[0 .. pathInfoStarts]; 1772 1773 auto question = requestUri.indexOf("?"); 1774 if(question == -1) { 1775 queryString = ""; 1776 // FIXME: double check, this might be wrong since it could be url encoded 1777 pathInfo = requestUri[pathInfoStarts..$]; 1778 } else { 1779 queryString = requestUri[question+1..$]; 1780 pathInfo = requestUri[pathInfoStarts..question]; 1781 } 1782 1783 auto ugh = decodeVariables(queryString, "&", &allGetNamesInOrder, &allGetValuesInOrder); 1784 getArray = cast(string[][string]) assumeUnique(ugh); 1785 1786 if(header.indexOf("HTTP/1.0") != -1) { 1787 http10 = true; 1788 autoBuffer = true; 1789 if(closeConnection) { 1790 // on http 1.0, close is assumed (unlike http/1.1 where we assume keep alive) 1791 *closeConnection = true; 1792 } 1793 } 1794 } else { 1795 // other header 1796 auto colon = header.indexOf(":"); 1797 if(colon == -1) 1798 throw new Exception("HTTP headers should have a colon!"); 1799 string name = header[0..colon].toLower; 1800 string value = header[colon+2..$]; // skip the colon and the space 1801 1802 requestHeadersHere[name] = value; 1803 1804 if (name == "accept") { 1805 accept = value; 1806 } 1807 else if (name == "origin") { 1808 origin = value; 1809 } 1810 else if (name == "connection") { 1811 if(value == "close" && closeConnection) 1812 *closeConnection = true; 1813 if(value.asLowerCase().canFind("keep-alive")) { 1814 keepAliveRequested = true; 1815 1816 // on http 1.0, the connection is closed by default, 1817 // but not if they request keep-alive. then we don't close 1818 // anymore - undoing the set above 1819 if(http10 && closeConnection) { 1820 *closeConnection = false; 1821 } 1822 } 1823 } 1824 else if (name == "transfer-encoding") { 1825 if(value == "chunked") 1826 isChunked = true; 1827 } 1828 else if (name == "last-event-id") { 1829 lastEventId = value; 1830 } 1831 else if (name == "authorization") { 1832 authorization = value; 1833 } 1834 else if (name == "content-type") { 1835 contentType = value; 1836 } 1837 else if (name == "content-length") { 1838 contentLength = to!size_t(value); 1839 } 1840 else if (name == "x-forwarded-for") { 1841 remoteAddress = value; 1842 } 1843 else if (name == "x-forwarded-host" || name == "host") { 1844 if(name != "host" || host is null) 1845 host = value; 1846 } 1847 // FIXME: https://tools.ietf.org/html/rfc7239 1848 else if (name == "accept-encoding") { 1849 if(value.indexOf("gzip") != -1) 1850 acceptsGzip = true; 1851 } 1852 else if (name == "user-agent") { 1853 userAgent = value; 1854 } 1855 else if (name == "referer") { 1856 referrer = value; 1857 } 1858 else if (name == "cookie") { 1859 cookie ~= value; 1860 } else if(name == "expect") { 1861 if(value == "100-continue") { 1862 // FIXME we should probably give user code a chance 1863 // to process and reject but that needs to be virtual, 1864 // perhaps part of the CGI redesign. 1865 1866 // FIXME: if size is > max content length it should 1867 // also fail at this point. 1868 _rawDataOutput(cast(ubyte[]) "HTTP/1.1 100 Continue\r\n\r\n"); 1869 1870 // FIXME: let the user write out 103 early hints too 1871 } 1872 } 1873 // else 1874 // ignore it 1875 1876 } 1877 } 1878 1879 inputData.consume(idx + 4); 1880 // done 1881 1882 requestHeaders = assumeUnique(requestHeadersHere); 1883 1884 ByChunkRange dataByChunk; 1885 1886 // reading Content-Length type data 1887 // We need to read up the data we have, and write it out as a chunk. 1888 if(!isChunked) { 1889 dataByChunk = byChunk(inputData, contentLength); 1890 } else { 1891 // chunked requests happen, but not every day. Since we need to know 1892 // the content length (for now, maybe that should change), we'll buffer 1893 // the whole thing here instead of parse streaming. (I think this is what Apache does anyway in cgi modes) 1894 auto data = dechunk(inputData); 1895 1896 // set the range here 1897 dataByChunk = byChunk(data); 1898 contentLength = data.length; 1899 } 1900 1901 assert(dataByChunk !is null); 1902 1903 if(contentLength) { 1904 prepareForIncomingDataChunks(contentType, contentLength); 1905 foreach(dataChunk; dataByChunk) { 1906 handleIncomingDataChunk(dataChunk); 1907 } 1908 postArray = assumeUnique(pps._post); 1909 filesArray = assumeUnique(pps._files); 1910 files = keepLastOf(filesArray); 1911 post = keepLastOf(postArray); 1912 postBody = pps.postBody; 1913 cleanUpPostDataState(); 1914 } 1915 1916 this.port = port; 1917 this.referrer = referrer; 1918 this.remoteAddress = remoteAddress; 1919 this.userAgent = userAgent; 1920 this.authorization = authorization; 1921 this.origin = origin; 1922 this.accept = accept; 1923 this.lastEventId = lastEventId; 1924 this.https = https; 1925 this.host = host; 1926 this.requestMethod = requestMethod; 1927 this.requestUri = requestUri; 1928 this.pathInfo = pathInfo; 1929 this.queryString = queryString; 1930 1931 this.scriptName = scriptName; 1932 this.get = keepLastOf(getArray); 1933 this.getArray = cast(immutable) getArray; 1934 this.keepAliveRequested = keepAliveRequested; 1935 this.acceptsGzip = acceptsGzip; 1936 this.cookie = cookie; 1937 1938 cookiesArray = getCookieArray(); 1939 cookies = keepLastOf(cookiesArray); 1940 1941 } 1942 BufferedInputRange idlol; 1943 1944 private immutable(string[string]) keepLastOf(in string[][string] arr) { 1945 string[string] ca; 1946 foreach(k, v; arr) 1947 ca[k] = v[$-1]; 1948 1949 return assumeUnique(ca); 1950 } 1951 1952 // FIXME duplication 1953 private immutable(UploadedFile[string]) keepLastOf(in UploadedFile[][string] arr) { 1954 UploadedFile[string] ca; 1955 foreach(k, v; arr) 1956 ca[k] = v[$-1]; 1957 1958 return assumeUnique(ca); 1959 } 1960 1961 1962 private immutable(string[][string]) getCookieArray() { 1963 auto forTheLoveOfGod = decodeVariables(cookie, "; "); 1964 return assumeUnique(forTheLoveOfGod); 1965 } 1966 1967 /// Very simple method to require a basic auth username and password. 1968 /// If the http request doesn't include the required credentials, it throws a 1969 /// HTTP 401 error, and an exception. 1970 /// 1971 /// Note: basic auth does not provide great security, especially over unencrypted HTTP; 1972 /// the user's credentials are sent in plain text on every request. 1973 /// 1974 /// If you are using Apache, the HTTP_AUTHORIZATION variable may not be sent to the 1975 /// application. Either use Apache's built in methods for basic authentication, or add 1976 /// something along these lines to your server configuration: 1977 /// 1978 /// RewriteEngine On 1979 /// RewriteCond %{HTTP:Authorization} ^(.*) 1980 /// RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1] 1981 /// 1982 /// To ensure the necessary data is available to cgi.d. 1983 void requireBasicAuth(string user, string pass, string message = null) { 1984 if(authorization != "Basic " ~ Base64.encode(cast(immutable(ubyte)[]) (user ~ ":" ~ pass))) { 1985 setResponseStatus("401 Authorization Required"); 1986 header ("WWW-Authenticate: Basic realm=\""~message~"\""); 1987 close(); 1988 throw new Exception("Not authorized; got " ~ authorization); 1989 } 1990 } 1991 1992 /// Very simple caching controls - setCache(false) means it will never be cached. Good for rapidly updated or sensitive sites. 1993 /// setCache(true) means it will always be cached for as long as possible. Best for static content. 1994 /// Use setResponseExpires and updateResponseExpires for more control 1995 void setCache(bool allowCaching) { 1996 noCache = !allowCaching; 1997 } 1998 1999 /// Set to true and use cgi.write(data, true); to send a gzipped response to browsers 2000 /// who can accept it 2001 bool gzipResponse; 2002 2003 immutable bool acceptsGzip; 2004 immutable bool keepAliveRequested; 2005 2006 /// Set to true if and only if this was initialized with command line arguments 2007 immutable bool isCalledWithCommandLineArguments; 2008 2009 /// This gets a full url for the current request, including port, protocol, host, path, and query 2010 string getCurrentCompleteUri() const { 2011 ushort defaultPort = https ? 443 : 80; 2012 2013 string uri = "http"; 2014 if(https) 2015 uri ~= "s"; 2016 uri ~= "://"; 2017 uri ~= host; 2018 /+ // the host has the port so p sure this never needed, cgi on apache and embedded http all do the right hting now 2019 version(none) 2020 if(!(!port || port == defaultPort)) { 2021 uri ~= ":"; 2022 uri ~= to!string(port); 2023 } 2024 +/ 2025 uri ~= requestUri; 2026 return uri; 2027 } 2028 2029 /// You can override this if your site base url isn't the same as the script name 2030 string logicalScriptName() const { 2031 return scriptName; 2032 } 2033 2034 /++ 2035 Sets the HTTP status of the response. For example, "404 File Not Found" or "500 Internal Server Error". 2036 It assumes "200 OK", and automatically changes to "302 Found" if you call setResponseLocation(). 2037 Note setResponseStatus() must be called *before* you write() any data to the output. 2038 2039 History: 2040 The `int` overload was added on January 11, 2021. 2041 +/ 2042 void setResponseStatus(string status) { 2043 assert(!outputtedResponseData); 2044 responseStatus = status; 2045 } 2046 /// ditto 2047 void setResponseStatus(int statusCode) { 2048 setResponseStatus(getHttpCodeText(statusCode)); 2049 } 2050 private string responseStatus = null; 2051 2052 /// Returns true if it is still possible to output headers 2053 bool canOutputHeaders() { 2054 return !isClosed && !outputtedResponseData; 2055 } 2056 2057 /// Sets the location header, which the browser will redirect the user to automatically. 2058 /// Note setResponseLocation() must be called *before* you write() any data to the output. 2059 /// The optional important argument is used if it's a default suggestion rather than something to insist upon. 2060 void setResponseLocation(string uri, bool important = true, string status = null) { 2061 if(!important && isCurrentResponseLocationImportant) 2062 return; // important redirects always override unimportant ones 2063 2064 if(uri is null) { 2065 responseStatus = "200 OK"; 2066 responseLocation = null; 2067 isCurrentResponseLocationImportant = important; 2068 return; // this just cancels the redirect 2069 } 2070 2071 assert(!outputtedResponseData); 2072 if(status is null) 2073 responseStatus = "302 Found"; 2074 else 2075 responseStatus = status; 2076 2077 responseLocation = uri.strip; 2078 isCurrentResponseLocationImportant = important; 2079 } 2080 protected string responseLocation = null; 2081 private bool isCurrentResponseLocationImportant = false; 2082 2083 /// Sets the Expires: http header. See also: updateResponseExpires, setPublicCaching 2084 /// The parameter is in unix_timestamp * 1000. Try setResponseExpires(getUTCtime() + SOME AMOUNT) for normal use. 2085 /// Note: the when parameter is different than setCookie's expire parameter. 2086 void setResponseExpires(long when, bool isPublic = false) { 2087 responseExpires = when; 2088 setCache(true); // need to enable caching so the date has meaning 2089 2090 responseIsPublic = isPublic; 2091 responseExpiresRelative = false; 2092 } 2093 2094 /// Sets a cache-control max-age header for whenFromNow, in seconds. 2095 void setResponseExpiresRelative(int whenFromNow, bool isPublic = false) { 2096 responseExpires = whenFromNow; 2097 setCache(true); // need to enable caching so the date has meaning 2098 2099 responseIsPublic = isPublic; 2100 responseExpiresRelative = true; 2101 } 2102 private long responseExpires = long.min; 2103 private bool responseIsPublic = false; 2104 private bool responseExpiresRelative = false; 2105 2106 /// This is like setResponseExpires, but it can be called multiple times. The setting most in the past is the one kept. 2107 /// If you have multiple functions, they all might call updateResponseExpires about their own return value. The program 2108 /// output as a whole is as cacheable as the least cachable part in the chain. 2109 2110 /// setCache(false) always overrides this - it is, by definition, the strictest anti-cache statement available. If your site outputs sensitive user data, you should probably call setCache(false) when you do, to ensure no other functions will cache the content, as it may be a privacy risk. 2111 /// Conversely, setting here overrides setCache(true), since any expiration date is in the past of infinity. 2112 void updateResponseExpires(long when, bool isPublic) { 2113 if(responseExpires == long.min) 2114 setResponseExpires(when, isPublic); 2115 else if(when < responseExpires) 2116 setResponseExpires(when, responseIsPublic && isPublic); // if any part of it is private, it all is 2117 } 2118 2119 /* 2120 /// Set to true if you want the result to be cached publically - that is, is the content shared? 2121 /// Should generally be false if the user is logged in. It assumes private cache only. 2122 /// setCache(true) also turns on public caching, and setCache(false) sets to private. 2123 void setPublicCaching(bool allowPublicCaches) { 2124 publicCaching = allowPublicCaches; 2125 } 2126 private bool publicCaching = false; 2127 */ 2128 2129 /++ 2130 History: 2131 Added January 11, 2021 2132 +/ 2133 enum SameSitePolicy { 2134 Lax, 2135 Strict, 2136 None 2137 } 2138 2139 /++ 2140 Sets an HTTP cookie, automatically encoding the data to the correct string. 2141 expiresIn is how many milliseconds in the future the cookie will expire. 2142 TIP: to make a cookie accessible from subdomains, set the domain to .yourdomain.com. 2143 Note setCookie() must be called *before* you write() any data to the output. 2144 2145 History: 2146 Parameter `sameSitePolicy` was added on January 11, 2021. 2147 +/ 2148 void setCookie(string name, string data, long expiresIn = 0, string path = null, string domain = null, bool httpOnly = false, bool secure = false, SameSitePolicy sameSitePolicy = SameSitePolicy.Lax) { 2149 assert(!outputtedResponseData); 2150 string cookie = std.uri.encodeComponent(name) ~ "="; 2151 cookie ~= std.uri.encodeComponent(data); 2152 if(path !is null) 2153 cookie ~= "; path=" ~ path; 2154 // FIXME: should I just be using max-age here? (also in cache below) 2155 if(expiresIn != 0) 2156 cookie ~= "; expires=" ~ printDate(cast(DateTime) Clock.currTime(UTC()) + dur!"msecs"(expiresIn)); 2157 if(domain !is null) 2158 cookie ~= "; domain=" ~ domain; 2159 if(secure == true) 2160 cookie ~= "; Secure"; 2161 if(httpOnly == true ) 2162 cookie ~= "; HttpOnly"; 2163 final switch(sameSitePolicy) { 2164 case SameSitePolicy.Lax: 2165 cookie ~= "; SameSite=Lax"; 2166 break; 2167 case SameSitePolicy.Strict: 2168 cookie ~= "; SameSite=Strict"; 2169 break; 2170 case SameSitePolicy.None: 2171 cookie ~= "; SameSite=None"; 2172 assert(secure); // cookie spec requires this now, see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite 2173 break; 2174 } 2175 2176 if(auto idx = name in cookieIndexes) { 2177 responseCookies[*idx] = cookie; 2178 } else { 2179 cookieIndexes[name] = responseCookies.length; 2180 responseCookies ~= cookie; 2181 } 2182 } 2183 private string[] responseCookies; 2184 private size_t[string] cookieIndexes; 2185 2186 /// Clears a previously set cookie with the given name, path, and domain. 2187 void clearCookie(string name, string path = null, string domain = null) { 2188 assert(!outputtedResponseData); 2189 setCookie(name, "", 1, path, domain); 2190 } 2191 2192 /// Sets the content type of the response, for example "text/html" (the default) for HTML, or "image/png" for a PNG image 2193 void setResponseContentType(string ct) { 2194 assert(!outputtedResponseData); 2195 responseContentType = ct; 2196 } 2197 private string responseContentType = null; 2198 2199 /// Adds a custom header. It should be the name: value, but without any line terminator. 2200 /// For example: header("X-My-Header: Some value"); 2201 /// Note you should use the specialized functions in this object if possible to avoid 2202 /// duplicates in the output. 2203 void header(string h) { 2204 customHeaders ~= h; 2205 } 2206 2207 /++ 2208 I named the original function `header` after PHP, but this pattern more fits 2209 the rest of the Cgi object. 2210 2211 Either name are allowed. 2212 2213 History: 2214 Alias added June 17, 2022. 2215 +/ 2216 alias setResponseHeader = header; 2217 2218 private string[] customHeaders; 2219 private bool websocketMode; 2220 2221 void flushHeaders(const(void)[] t, bool isAll = false) { 2222 StackBuffer buffer = StackBuffer(0); 2223 2224 prepHeaders(t, isAll, &buffer); 2225 2226 if(rawDataOutput !is null) 2227 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2228 else { 2229 stdout.rawWrite(buffer.get()); 2230 } 2231 } 2232 2233 private void prepHeaders(const(void)[] t, bool isAll, StackBuffer* buffer) { 2234 string terminator = "\n"; 2235 if(rawDataOutput !is null) 2236 terminator = "\r\n"; 2237 2238 if(responseStatus !is null) { 2239 if(nph) { 2240 if(http10) 2241 buffer.add("HTTP/1.0 ", responseStatus, terminator); 2242 else 2243 buffer.add("HTTP/1.1 ", responseStatus, terminator); 2244 } else 2245 buffer.add("Status: ", responseStatus, terminator); 2246 } else if (nph) { 2247 if(http10) 2248 buffer.add("HTTP/1.0 200 OK", terminator); 2249 else 2250 buffer.add("HTTP/1.1 200 OK", terminator); 2251 } 2252 2253 if(websocketMode) 2254 goto websocket; 2255 2256 if(nph) { // we're responsible for setting the date too according to http 1.1 2257 char[29] db = void; 2258 printDateToBuffer(cast(DateTime) Clock.currTime(UTC()), db[]); 2259 buffer.add("Date: ", db[], terminator); 2260 } 2261 2262 // FIXME: what if the user wants to set his own content-length? 2263 // The custom header function can do it, so maybe that's best. 2264 // Or we could reuse the isAll param. 2265 if(responseLocation !is null) { 2266 buffer.add("Location: ", responseLocation, terminator); 2267 } 2268 if(!noCache && responseExpires != long.min) { // an explicit expiration date is set 2269 if(responseExpiresRelative) { 2270 buffer.add("Cache-Control: ", responseIsPublic ? "public" : "private", ", max-age="); 2271 buffer.add(responseExpires); 2272 buffer.add(", no-cache=\"set-cookie, set-cookie2\"", terminator); 2273 } else { 2274 auto expires = SysTime(unixTimeToStdTime(cast(int)(responseExpires / 1000)), UTC()); 2275 char[29] db = void; 2276 printDateToBuffer(cast(DateTime) expires, db[]); 2277 buffer.add("Expires: ", db[], terminator); 2278 // FIXME: assuming everything is private unless you use nocache - generally right for dynamic pages, but not necessarily 2279 buffer.add("Cache-Control: ", (responseIsPublic ? "public" : "private"), ", no-cache=\"set-cookie, set-cookie2\""); 2280 buffer.add(terminator); 2281 } 2282 } 2283 if(responseCookies !is null && responseCookies.length > 0) { 2284 foreach(c; responseCookies) 2285 buffer.add("Set-Cookie: ", c, terminator); 2286 } 2287 if(noCache) { // we specifically do not want caching (this is actually the default) 2288 buffer.add("Cache-Control: private, no-cache=\"set-cookie\"", terminator); 2289 buffer.add("Expires: 0", terminator); 2290 buffer.add("Pragma: no-cache", terminator); 2291 } else { 2292 if(responseExpires == long.min) { // caching was enabled, but without a date set - that means assume cache forever 2293 buffer.add("Cache-Control: public", terminator); 2294 buffer.add("Expires: Tue, 31 Dec 2030 14:00:00 GMT", terminator); // FIXME: should not be more than one year in the future 2295 } 2296 } 2297 if(responseContentType !is null) { 2298 buffer.add("Content-Type: ", responseContentType, terminator); 2299 } else 2300 buffer.add("Content-Type: text/html; charset=utf-8", terminator); 2301 2302 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2303 buffer.add("Content-Encoding: gzip", terminator); 2304 } 2305 2306 2307 if(!isAll) { 2308 if(nph && !http10) { 2309 buffer.add("Transfer-Encoding: chunked", terminator); 2310 responseChunked = true; 2311 } 2312 } else { 2313 buffer.add("Content-Length: "); 2314 buffer.add(t.length); 2315 buffer.add(terminator); 2316 if(nph && keepAliveRequested) { 2317 buffer.add("Connection: Keep-Alive", terminator); 2318 } 2319 } 2320 2321 websocket: 2322 2323 foreach(hd; customHeaders) 2324 buffer.add(hd, terminator); 2325 2326 // FIXME: what about duplicated headers? 2327 2328 // end of header indicator 2329 buffer.add(terminator); 2330 2331 outputtedResponseData = true; 2332 } 2333 2334 /// Writes the data to the output, flushing headers if they have not yet been sent. 2335 void write(const(void)[] t, bool isAll = false, bool maybeAutoClose = true) { 2336 assert(!closed, "Output has already been closed"); 2337 2338 StackBuffer buffer = StackBuffer(0); 2339 2340 if(gzipResponse && acceptsGzip && isAll) { // FIXME: isAll really shouldn't be necessary 2341 // actually gzip the data here 2342 2343 auto c = new Compress(HeaderFormat.gzip); // want gzip 2344 2345 auto data = c.compress(t); 2346 data ~= c.flush(); 2347 2348 // std.file.write("/tmp/last-item", data); 2349 2350 t = data; 2351 } 2352 2353 if(!outputtedResponseData && (!autoBuffer || isAll)) { 2354 prepHeaders(t, isAll, &buffer); 2355 } 2356 2357 if(requestMethod != RequestMethod.HEAD && t.length > 0) { 2358 if (autoBuffer && !isAll) { 2359 outputBuffer ~= cast(ubyte[]) t; 2360 } 2361 if(!autoBuffer || isAll) { 2362 if(rawDataOutput !is null) 2363 if(nph && responseChunked) { 2364 //rawDataOutput(makeChunk(cast(const(ubyte)[]) t)); 2365 // we're making the chunk here instead of in a function 2366 // to avoid unneeded gc pressure 2367 buffer.add(toHex(t.length)); 2368 buffer.add("\r\n"); 2369 buffer.add(cast(char[]) t, "\r\n"); 2370 } else { 2371 buffer.add(cast(char[]) t); 2372 } 2373 else 2374 buffer.add(cast(char[]) t); 2375 } 2376 } 2377 2378 if(rawDataOutput !is null) 2379 rawDataOutput(cast(const(ubyte)[]) buffer.get()); 2380 else 2381 stdout.rawWrite(buffer.get()); 2382 2383 if(maybeAutoClose && isAll) 2384 close(); // if you say it is all, that means we're definitely done 2385 // maybeAutoClose can be false though to avoid this (important if you call from inside close()! 2386 } 2387 2388 /++ 2389 Convenience method to set content type to json and write the string as the complete response. 2390 2391 History: 2392 Added January 16, 2020 2393 +/ 2394 void writeJson(string json) { 2395 this.setResponseContentType("application/json"); 2396 this.write(json, true); 2397 } 2398 2399 /// Flushes the pending buffer, leaving the connection open so you can send more. 2400 void flush() { 2401 if(rawDataOutput is null) 2402 stdout.flush(); 2403 else if(flushDelegate !is null) 2404 flushDelegate(); 2405 } 2406 2407 version(autoBuffer) 2408 bool autoBuffer = true; 2409 else 2410 bool autoBuffer = false; 2411 ubyte[] outputBuffer; 2412 2413 /// Flushes the buffers to the network, signifying that you are done. 2414 /// You should always call this explicitly when you are done outputting data. 2415 void close() { 2416 if(closed) 2417 return; // don't double close 2418 2419 if(!outputtedResponseData) 2420 write("", true, false); 2421 2422 // writing auto buffered data 2423 if(requestMethod != RequestMethod.HEAD && autoBuffer) { 2424 if(!nph) 2425 stdout.rawWrite(outputBuffer); 2426 else 2427 write(outputBuffer, true, false); // tell it this is everything 2428 } 2429 2430 // closing the last chunk... 2431 if(nph && rawDataOutput !is null && responseChunked) 2432 rawDataOutput(cast(const(ubyte)[]) "0\r\n\r\n"); 2433 2434 if(flushDelegate) 2435 flushDelegate(); 2436 2437 closed = true; 2438 } 2439 2440 // Closes without doing anything, shouldn't be used often 2441 void rawClose() { 2442 closed = true; 2443 } 2444 2445 /++ 2446 Gets a request variable as a specific type, or the default value of it isn't there 2447 or isn't convertible to the request type. 2448 2449 Checks both GET and POST variables, preferring the POST variable, if available. 2450 2451 A nice trick is using the default value to choose the type: 2452 2453 --- 2454 /* 2455 The return value will match the type of the default. 2456 Here, I gave 10 as a default, so the return value will 2457 be an int. 2458 2459 If the user-supplied value cannot be converted to the 2460 requested type, you will get the default value back. 2461 */ 2462 int a = cgi.request("number", 10); 2463 2464 if(cgi.get["number"] == "11") 2465 assert(a == 11); // conversion succeeds 2466 2467 if("number" !in cgi.get) 2468 assert(a == 10); // no value means you can't convert - give the default 2469 2470 if(cgi.get["number"] == "twelve") 2471 assert(a == 10); // conversion from string to int would fail, so we get the default 2472 --- 2473 2474 You can use an enum as an easy whitelist, too: 2475 2476 --- 2477 enum Operations { 2478 add, remove, query 2479 } 2480 2481 auto op = cgi.request("op", Operations.query); 2482 2483 if(cgi.get["op"] == "add") 2484 assert(op == Operations.add); 2485 if(cgi.get["op"] == "remove") 2486 assert(op == Operations.remove); 2487 if(cgi.get["op"] == "query") 2488 assert(op == Operations.query); 2489 2490 if(cgi.get["op"] == "random string") 2491 assert(op == Operations.query); // the value can't be converted to the enum, so we get the default 2492 --- 2493 +/ 2494 T request(T = string)(in string name, in T def = T.init) const nothrow { 2495 try { 2496 return 2497 (name in post) ? to!T(post[name]) : 2498 (name in get) ? to!T(get[name]) : 2499 def; 2500 } catch(Exception e) { return def; } 2501 } 2502 2503 /// Is the output already closed? 2504 bool isClosed() const { 2505 return closed; 2506 } 2507 2508 /++ 2509 Gets a session object associated with the `cgi` request. You can use different type throughout your application. 2510 +/ 2511 Session!Data getSessionObject(Data)() { 2512 if(testInProcess !is null) { 2513 // test mode 2514 auto obj = testInProcess.getSessionOverride(typeid(typeof(return))); 2515 if(obj !is null) 2516 return cast(typeof(return)) obj; 2517 else { 2518 auto o = new MockSession!Data(); 2519 testInProcess.setSessionOverride(typeid(typeof(return)), o); 2520 return o; 2521 } 2522 } else { 2523 // normal operation 2524 return new BasicDataServerSession!Data(this); 2525 } 2526 } 2527 2528 // if it is in test mode; triggers mock sessions. Used by CgiTester 2529 version(with_breaking_cgi_features) 2530 private CgiTester testInProcess; 2531 2532 /* Hooks for redirecting input and output */ 2533 private void delegate(const(ubyte)[]) rawDataOutput = null; 2534 private void delegate() flushDelegate = null; 2535 2536 /* This info is used when handling a more raw HTTP protocol */ 2537 private bool nph; 2538 private bool http10; 2539 private bool closed; 2540 private bool responseChunked = false; 2541 2542 version(preserveData) // note: this can eat lots of memory; don't use unless you're sure you need it. 2543 immutable(ubyte)[] originalPostData; 2544 2545 public immutable string postBody; 2546 alias postJson = postBody; // old name 2547 2548 /* Internal state flags */ 2549 private bool outputtedResponseData; 2550 private bool noCache = true; 2551 2552 const(string[string]) environmentVariables; 2553 2554 /** What follows is data gotten from the HTTP request. It is all fully immutable, 2555 partially because it logically is (your code doesn't change what the user requested...) 2556 and partially because I hate how bad programs in PHP change those superglobals to do 2557 all kinds of hard to follow ugliness. I don't want that to ever happen in D. 2558 2559 For some of these, you'll want to refer to the http or cgi specs for more details. 2560 */ 2561 immutable(string[string]) requestHeaders; /// All the raw headers in the request as name/value pairs. The name is stored as all lower case, but otherwise the same as it is in HTTP; words separated by dashes. For example, "cookie" or "accept-encoding". Many HTTP headers have specialized variables below for more convenience and static name checking; you should generally try to use them. 2562 2563 immutable(char[]) host; /// The hostname in the request. If one program serves multiple domains, you can use this to differentiate between them. 2564 immutable(char[]) origin; /// The origin header in the request, if present. Some HTML5 cross-domain apis set this and you should check it on those cross domain requests and websockets. 2565 immutable(char[]) userAgent; /// The browser's user-agent string. Can be used to identify the browser. 2566 immutable(char[]) pathInfo; /// This is any stuff sent after your program's name on the url, but before the query string. For example, suppose your program is named "app". If the user goes to site.com/app, pathInfo is empty. But, he can also go to site.com/app/some/sub/path; treating your program like a virtual folder. In this case, pathInfo == "/some/sub/path". 2567 immutable(char[]) scriptName; /// The full base path of your program, as seen by the user. If your program is located at site.com/programs/apps, scriptName == "/programs/apps". 2568 immutable(char[]) scriptFileName; /// The physical filename of your script 2569 immutable(char[]) authorization; /// The full authorization string from the header, undigested. Useful for implementing auth schemes such as OAuth 1.0. Note that some web servers do not forward this to the app without taking extra steps. See requireBasicAuth's comment for more info. 2570 immutable(char[]) accept; /// The HTTP accept header is the user agent telling what content types it is willing to accept. This is often */*; they accept everything, so it's not terribly useful. (The similar sounding Accept-Encoding header is handled automatically for chunking and gzipping. Simply set gzipResponse = true and cgi.d handles the details, zipping if the user's browser is willing to accept it.) 2571 immutable(char[]) lastEventId; /// The HTML 5 draft includes an EventSource() object that connects to the server, and remains open to take a stream of events. My arsd.rtud module can help with the server side part of that. The Last-Event-Id http header is defined in the draft to help handle loss of connection. When the browser reconnects to you, it sets this header to the last event id it saw, so you can catch it up. This member has the contents of that header. 2572 2573 immutable(RequestMethod) requestMethod; /// The HTTP request verb: GET, POST, etc. It is represented as an enum in cgi.d (which, like many enums, you can convert back to string with std.conv.to()). A HTTP GET is supposed to, according to the spec, not have side effects; a user can GET something over and over again and always have the same result. On all requests, the get[] and getArray[] members may be filled in. The post[] and postArray[] members are only filled in on POST methods. 2574 immutable(char[]) queryString; /// The unparsed content of the request query string - the stuff after the ? in your URL. See get[] and getArray[] for a parse view of it. Sometimes, the unparsed string is useful though if you want a custom format of data up there (probably not a good idea, unless it is really simple, like "?username" perhaps.) 2575 immutable(char[]) cookie; /// The unparsed content of the Cookie: header in the request. See also the cookies[string] member for a parsed view of the data. 2576 /** The Referer header from the request. (It is misspelled in the HTTP spec, and thus the actual request and cgi specs too, but I spelled the word correctly here because that's sane. The spec's misspelling is an implementation detail.) It contains the site url that referred the user to your program; the site that linked to you, or if you're serving images, the site that has you as an image. Also, if you're in an iframe, the referrer is the site that is framing you. 2577 2578 Important note: if the user copy/pastes your url, this is blank, and, just like with all other user data, their browsers can also lie to you. Don't rely on it for real security. 2579 */ 2580 immutable(char[]) referrer; 2581 immutable(char[]) requestUri; /// The full url if the current request, excluding the protocol and host. requestUri == scriptName ~ pathInfo ~ (queryString.length ? "?" ~ queryString : ""); 2582 2583 immutable(char[]) remoteAddress; /// The IP address of the user, as we see it. (Might not match the IP of the user's computer due to things like proxies and NAT.) 2584 2585 immutable bool https; /// Was the request encrypted via https? 2586 immutable int port; /// On what TCP port number did the server receive the request? 2587 2588 /** Here come the parsed request variables - the things that come close to PHP's _GET, _POST, etc. superglobals in content. */ 2589 2590 immutable(string[string]) get; /// The data from your query string in the url, only showing the last string of each name. If you want to handle multiple values with the same name, use getArray. This only works right if the query string is x-www-form-urlencoded; the default you see on the web with name=value pairs separated by the & character. 2591 immutable(string[string]) post; /// The data from the request's body, on POST requests. It parses application/x-www-form-urlencoded data (used by most web requests, including typical forms), and multipart/form-data requests (used by file uploads on web forms) into the same container, so you can always access them the same way. It makes no attempt to parse other content types. If you want to accept an XML Post body (for a web api perhaps), you'll need to handle the raw data yourself. 2592 immutable(string[string]) cookies; /// Separates out the cookie header into individual name/value pairs (which is how you set them!) 2593 2594 /** 2595 Represents user uploaded files. 2596 2597 When making a file upload form, be sure to follow the standard: set method="POST" and enctype="multipart/form-data" in your html <form> tag attributes. The key into this array is the name attribute on your input tag, just like with other post variables. See the comments on the UploadedFile struct for more information about the data inside, including important notes on max size and content location. 2598 */ 2599 immutable(UploadedFile[][string]) filesArray; 2600 immutable(UploadedFile[string]) files; 2601 2602 /// Use these if you expect multiple items submitted with the same name. btw, assert(get[name] is getArray[name][$-1); should pass. Same for post and cookies. 2603 /// the order of the arrays is the order the data arrives 2604 immutable(string[][string]) getArray; /// like get, but an array of values per name 2605 immutable(string[][string]) postArray; /// ditto for post 2606 immutable(string[][string]) cookiesArray; /// ditto for cookies 2607 2608 // convenience function for appending to a uri without extra ? 2609 // matches the name and effect of javascript's location.search property 2610 string search() const { 2611 if(queryString.length) 2612 return "?" ~ queryString; 2613 return ""; 2614 } 2615 2616 // FIXME: what about multiple files with the same name? 2617 private: 2618 //RequestMethod _requestMethod; 2619 } 2620 2621 /// use this for testing or other isolated things when you want it to be no-ops 2622 Cgi dummyCgi(Cgi.RequestMethod method = Cgi.RequestMethod.GET, string url = null, in ubyte[] data = null, void delegate(const(ubyte)[]) outputSink = null) { 2623 // we want to ignore, not use stdout 2624 if(outputSink is null) 2625 outputSink = delegate void(const(ubyte)[]) { }; 2626 2627 string[string] env; 2628 env["REQUEST_METHOD"] = to!string(method); 2629 env["CONTENT_LENGTH"] = to!string(data.length); 2630 2631 auto cgi = new Cgi( 2632 0, 2633 env, 2634 { return data; }, 2635 outputSink, 2636 null); 2637 2638 return cgi; 2639 } 2640 2641 /++ 2642 A helper test class for request handler unittests. 2643 +/ 2644 version(with_breaking_cgi_features) 2645 class CgiTester { 2646 private { 2647 SessionObject[TypeInfo] mockSessions; 2648 SessionObject getSessionOverride(TypeInfo ti) { 2649 if(auto o = ti in mockSessions) 2650 return *o; 2651 else 2652 return null; 2653 } 2654 void setSessionOverride(TypeInfo ti, SessionObject so) { 2655 mockSessions[ti] = so; 2656 } 2657 } 2658 2659 /++ 2660 Gets (and creates if necessary) a mock session object for this test. Note 2661 it will be the same one used for any test operations through this CgiTester instance. 2662 +/ 2663 Session!Data getSessionObject(Data)() { 2664 auto obj = getSessionOverride(typeid(typeof(return))); 2665 if(obj !is null) 2666 return cast(typeof(return)) obj; 2667 else { 2668 auto o = new MockSession!Data(); 2669 setSessionOverride(typeid(typeof(return)), o); 2670 return o; 2671 } 2672 } 2673 2674 /++ 2675 Pass a reference to your request handler when creating the tester. 2676 +/ 2677 this(void function(Cgi) requestHandler) { 2678 this.requestHandler = requestHandler; 2679 } 2680 2681 /++ 2682 You can check response information with these methods after you call the request handler. 2683 +/ 2684 struct Response { 2685 int code; 2686 string[string] headers; 2687 string responseText; 2688 ubyte[] responseBody; 2689 } 2690 2691 /++ 2692 Executes a test request on your request handler, and returns the response. 2693 2694 Params: 2695 url = The URL to test. Should be an absolute path, but excluding domain. e.g. `"/test"`. 2696 args = additional arguments. Same format as cgi's command line handler. 2697 +/ 2698 Response GET(string url, string[] args = null) { 2699 return executeTest("GET", url, args); 2700 } 2701 /// ditto 2702 Response POST(string url, string[] args = null) { 2703 return executeTest("POST", url, args); 2704 } 2705 2706 /// ditto 2707 Response executeTest(string method, string url, string[] args) { 2708 ubyte[] outputtedRawData; 2709 void outputSink(const(ubyte)[] data) { 2710 outputtedRawData ~= data; 2711 } 2712 auto cgi = new Cgi(["test", method, url] ~ args, &outputSink); 2713 cgi.testInProcess = this; 2714 scope(exit) cgi.dispose(); 2715 2716 requestHandler(cgi); 2717 2718 cgi.close(); 2719 2720 Response response; 2721 2722 if(outputtedRawData.length) { 2723 enum LINE = "\r\n"; 2724 2725 auto idx = outputtedRawData.locationOf(LINE ~ LINE); 2726 assert(idx != -1, to!string(outputtedRawData)); 2727 auto headers = cast(string) outputtedRawData[0 .. idx]; 2728 response.code = 200; 2729 while(headers.length) { 2730 auto i = headers.locationOf(LINE); 2731 if(i == -1) i = cast(int) headers.length; 2732 2733 auto header = headers[0 .. i]; 2734 2735 auto c = header.locationOf(":"); 2736 if(c != -1) { 2737 auto name = header[0 .. c]; 2738 auto value = header[c + 2 ..$]; 2739 2740 if(name == "Status") 2741 response.code = value[0 .. value.locationOf(" ")].to!int; 2742 2743 response.headers[name] = value; 2744 } else { 2745 assert(0); 2746 } 2747 2748 if(i != headers.length) 2749 i += 2; 2750 headers = headers[i .. $]; 2751 } 2752 response.responseBody = outputtedRawData[idx + 4 .. $]; 2753 response.responseText = cast(string) response.responseBody; 2754 } 2755 2756 return response; 2757 } 2758 2759 private void function(Cgi) requestHandler; 2760 } 2761 2762 2763 // should this be a separate module? Probably, but that's a hassle. 2764 2765 /// Makes a data:// uri that can be used as links in most newer browsers (IE8+). 2766 string makeDataUrl(string mimeType, in void[] data) { 2767 auto data64 = Base64.encode(cast(const(ubyte[])) data); 2768 return "data:" ~ mimeType ~ ";base64," ~ assumeUnique(data64); 2769 } 2770 2771 // FIXME: I don't think this class correctly decodes/encodes the individual parts 2772 /// Represents a url that can be broken down or built up through properties 2773 struct Uri { 2774 alias toString this; // blargh idk a url really is a string, but should it be implicit? 2775 2776 // scheme//userinfo@host:port/path?query#fragment 2777 2778 string scheme; /// e.g. "http" in "http://example.com/" 2779 string userinfo; /// the username (and possibly a password) in the uri 2780 string host; /// the domain name 2781 int port; /// port number, if given. Will be zero if a port was not explicitly given 2782 string path; /// e.g. "/folder/file.html" in "http://example.com/folder/file.html" 2783 string query; /// the stuff after the ? in a uri 2784 string fragment; /// the stuff after the # in a uri. 2785 2786 // idk if i want to keep these, since the functions they wrap are used many, many, many times in existing code, so this is either an unnecessary alias or a gratuitous break of compatibility 2787 // the decode ones need to keep different names anyway because we can't overload on return values... 2788 static string encode(string s) { return std.uri.encodeComponent(s); } 2789 static string encode(string[string] s) { return encodeVariables(s); } 2790 static string encode(string[][string] s) { return encodeVariables(s); } 2791 2792 /// Breaks down a uri string to its components 2793 this(string uri) { 2794 reparse(uri); 2795 } 2796 2797 private void reparse(string uri) { 2798 // from RFC 3986 2799 // the ctRegex triples the compile time and makes ugly errors for no real benefit 2800 // it was a nice experiment but just not worth it. 2801 // enum ctr = ctRegex!r"^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?"; 2802 /* 2803 Captures: 2804 0 = whole url 2805 1 = scheme, with : 2806 2 = scheme, no : 2807 3 = authority, with // 2808 4 = authority, no // 2809 5 = path 2810 6 = query string, with ? 2811 7 = query string, no ? 2812 8 = anchor, with # 2813 9 = anchor, no # 2814 */ 2815 // Yikes, even regular, non-CT regex is also unacceptably slow to compile. 1.9s on my computer! 2816 // instead, I will DIY and cut that down to 0.6s on the same computer. 2817 /* 2818 2819 Note that authority is 2820 user:password@domain:port 2821 where the user:password@ part is optional, and the :port is optional. 2822 2823 Regex translation: 2824 2825 Scheme cannot have :, /, ?, or # in it, and must have one or more chars and end in a :. It is optional, but must be first. 2826 Authority must start with //, but cannot have any other /, ?, or # in it. It is optional. 2827 Path cannot have any ? or # in it. It is optional. 2828 Query must start with ? and must not have # in it. It is optional. 2829 Anchor must start with # and can have anything else in it to end of string. It is optional. 2830 */ 2831 2832 this = Uri.init; // reset all state 2833 2834 // empty uri = nothing special 2835 if(uri.length == 0) { 2836 return; 2837 } 2838 2839 size_t idx; 2840 2841 scheme_loop: foreach(char c; uri[idx .. $]) { 2842 switch(c) { 2843 case ':': 2844 case '/': 2845 case '?': 2846 case '#': 2847 break scheme_loop; 2848 default: 2849 } 2850 idx++; 2851 } 2852 2853 if(idx == 0 && uri[idx] == ':') { 2854 // this is actually a path! we skip way ahead 2855 goto path_loop; 2856 } 2857 2858 if(idx == uri.length) { 2859 // the whole thing is a path, apparently 2860 path = uri; 2861 return; 2862 } 2863 2864 if(idx > 0 && uri[idx] == ':') { 2865 scheme = uri[0 .. idx]; 2866 idx++; 2867 } else { 2868 // we need to rewind; it found a / but no :, so the whole thing is prolly a path... 2869 idx = 0; 2870 } 2871 2872 if(idx + 2 < uri.length && uri[idx .. idx + 2] == "//") { 2873 // we have an authority.... 2874 idx += 2; 2875 2876 auto authority_start = idx; 2877 authority_loop: foreach(char c; uri[idx .. $]) { 2878 switch(c) { 2879 case '/': 2880 case '?': 2881 case '#': 2882 break authority_loop; 2883 default: 2884 } 2885 idx++; 2886 } 2887 2888 auto authority = uri[authority_start .. idx]; 2889 2890 auto idx2 = authority.indexOf("@"); 2891 if(idx2 != -1) { 2892 userinfo = authority[0 .. idx2]; 2893 authority = authority[idx2 + 1 .. $]; 2894 } 2895 2896 if(authority.length && authority[0] == '[') { 2897 // ipv6 address special casing 2898 idx2 = authority.indexOf(']'); 2899 if(idx2 != -1) { 2900 auto end = authority[idx2 + 1 .. $]; 2901 if(end.length && end[0] == ':') 2902 idx2 = idx2 + 1; 2903 else 2904 idx2 = -1; 2905 } 2906 } else { 2907 idx2 = authority.indexOf(":"); 2908 } 2909 2910 if(idx2 == -1) { 2911 port = 0; // 0 means not specified; we should use the default for the scheme 2912 host = authority; 2913 } else { 2914 host = authority[0 .. idx2]; 2915 if(idx2 + 1 < authority.length) 2916 port = to!int(authority[idx2 + 1 .. $]); 2917 else 2918 port = 0; 2919 } 2920 } 2921 2922 path_loop: 2923 auto path_start = idx; 2924 2925 foreach(char c; uri[idx .. $]) { 2926 if(c == '?' || c == '#') 2927 break; 2928 idx++; 2929 } 2930 2931 path = uri[path_start .. idx]; 2932 2933 if(idx == uri.length) 2934 return; // nothing more to examine... 2935 2936 if(uri[idx] == '?') { 2937 idx++; 2938 auto query_start = idx; 2939 foreach(char c; uri[idx .. $]) { 2940 if(c == '#') 2941 break; 2942 idx++; 2943 } 2944 query = uri[query_start .. idx]; 2945 } 2946 2947 if(idx < uri.length && uri[idx] == '#') { 2948 idx++; 2949 fragment = uri[idx .. $]; 2950 } 2951 2952 // uriInvalidated = false; 2953 } 2954 2955 private string rebuildUri() const { 2956 string ret; 2957 if(scheme.length) 2958 ret ~= scheme ~ ":"; 2959 if(userinfo.length || host.length) 2960 ret ~= "//"; 2961 if(userinfo.length) 2962 ret ~= userinfo ~ "@"; 2963 if(host.length) 2964 ret ~= host; 2965 if(port) 2966 ret ~= ":" ~ to!string(port); 2967 2968 ret ~= path; 2969 2970 if(query.length) 2971 ret ~= "?" ~ query; 2972 2973 if(fragment.length) 2974 ret ~= "#" ~ fragment; 2975 2976 // uri = ret; 2977 // uriInvalidated = false; 2978 return ret; 2979 } 2980 2981 /// Converts the broken down parts back into a complete string 2982 string toString() const { 2983 // if(uriInvalidated) 2984 return rebuildUri(); 2985 } 2986 2987 /// Returns a new absolute Uri given a base. It treats this one as 2988 /// relative where possible, but absolute if not. (If protocol, domain, or 2989 /// other info is not set, the new one inherits it from the base.) 2990 /// 2991 /// Browsers use a function like this to figure out links in html. 2992 Uri basedOn(in Uri baseUrl) const { 2993 Uri n = this; // copies 2994 if(n.scheme == "data") 2995 return n; 2996 // n.uriInvalidated = true; // make sure we regenerate... 2997 2998 // userinfo is not inherited... is this wrong? 2999 3000 // if anything is given in the existing url, we don't use the base anymore. 3001 if(n.scheme.empty) { 3002 n.scheme = baseUrl.scheme; 3003 if(n.host.empty) { 3004 n.host = baseUrl.host; 3005 if(n.port == 0) { 3006 n.port = baseUrl.port; 3007 if(n.path.length > 0 && n.path[0] != '/') { 3008 auto b = baseUrl.path[0 .. baseUrl.path.lastIndexOf("/") + 1]; 3009 if(b.length == 0) 3010 b = "/"; 3011 n.path = b ~ n.path; 3012 } else if(n.path.length == 0) { 3013 n.path = baseUrl.path; 3014 } 3015 } 3016 } 3017 } 3018 3019 n.removeDots(); 3020 3021 return n; 3022 } 3023 3024 void removeDots() { 3025 auto parts = this.path.split("/"); 3026 string[] toKeep; 3027 foreach(part; parts) { 3028 if(part == ".") { 3029 continue; 3030 } else if(part == "..") { 3031 //if(toKeep.length > 1) 3032 toKeep = toKeep[0 .. $-1]; 3033 //else 3034 //toKeep = [""]; 3035 continue; 3036 } else { 3037 //if(toKeep.length && toKeep[$-1].length == 0 && part.length == 0) 3038 //continue; // skip a `//` situation 3039 toKeep ~= part; 3040 } 3041 } 3042 3043 auto path = toKeep.join("/"); 3044 if(path.length && path[0] != '/') 3045 path = "/" ~ path; 3046 3047 this.path = path; 3048 } 3049 3050 unittest { 3051 auto uri = Uri("test.html"); 3052 assert(uri.path == "test.html"); 3053 uri = Uri("path/1/lol"); 3054 assert(uri.path == "path/1/lol"); 3055 uri = Uri("http://me@example.com"); 3056 assert(uri.scheme == "http"); 3057 assert(uri.userinfo == "me"); 3058 assert(uri.host == "example.com"); 3059 uri = Uri("http://example.com/#a"); 3060 assert(uri.scheme == "http"); 3061 assert(uri.host == "example.com"); 3062 assert(uri.fragment == "a"); 3063 uri = Uri("#foo"); 3064 assert(uri.fragment == "foo"); 3065 uri = Uri("?lol"); 3066 assert(uri.query == "lol"); 3067 uri = Uri("#foo?lol"); 3068 assert(uri.fragment == "foo?lol"); 3069 uri = Uri("?lol#foo"); 3070 assert(uri.fragment == "foo"); 3071 assert(uri.query == "lol"); 3072 3073 uri = Uri("http://127.0.0.1/"); 3074 assert(uri.host == "127.0.0.1"); 3075 assert(uri.port == 0); 3076 3077 uri = Uri("http://127.0.0.1:123/"); 3078 assert(uri.host == "127.0.0.1"); 3079 assert(uri.port == 123); 3080 3081 uri = Uri("http://[ff:ff::0]/"); 3082 assert(uri.host == "[ff:ff::0]"); 3083 3084 uri = Uri("http://[ff:ff::0]:123/"); 3085 assert(uri.host == "[ff:ff::0]"); 3086 assert(uri.port == 123); 3087 } 3088 3089 // This can sometimes be a big pain in the butt for me, so lots of copy/paste here to cover 3090 // the possibilities. 3091 unittest { 3092 auto url = Uri("cool.html"); // checking relative links 3093 3094 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/cool.html"); 3095 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/cool.html"); 3096 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/cool.html"); 3097 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/cool.html"); 3098 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3099 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/cool.html"); 3100 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/cool.html"); 3101 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/cool.html"); 3102 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/cool.html"); 3103 3104 url = Uri("/something/cool.html"); // same server, different path 3105 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/something/cool.html"); 3106 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/something/cool.html"); 3107 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/something/cool.html"); 3108 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/something/cool.html"); 3109 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3110 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/something/cool.html"); 3111 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/something/cool.html"); 3112 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/something/cool.html"); 3113 assert(url.basedOn(Uri("http://test.com")) == "http://test.com/something/cool.html"); 3114 3115 url = Uri("?query=answer"); // same path. server, protocol, and port, just different query string and fragment 3116 assert(url.basedOn(Uri("http://test.com/what/test.html")) == "http://test.com/what/test.html?query=answer"); 3117 assert(url.basedOn(Uri("https://test.com/what/test.html")) == "https://test.com/what/test.html?query=answer"); 3118 assert(url.basedOn(Uri("http://test.com/what/")) == "http://test.com/what/?query=answer"); 3119 assert(url.basedOn(Uri("http://test.com/")) == "http://test.com/?query=answer"); 3120 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3121 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b")) == "http://test.com/what/test.html?query=answer"); 3122 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d")) == "http://test.com/what/test.html?query=answer"); 3123 assert(url.basedOn(Uri("http://test.com/what/test.html?a=b&c=d#what")) == "http://test.com/what/test.html?query=answer"); 3124 assert(url.basedOn(Uri("http://test.com")) == "http://test.com?query=answer"); 3125 3126 url = Uri("/test/bar"); 3127 assert(Uri("./").basedOn(url) == "/test/", Uri("./").basedOn(url)); 3128 assert(Uri("../").basedOn(url) == "/"); 3129 3130 url = Uri("http://example.com/"); 3131 assert(Uri("../foo").basedOn(url) == "http://example.com/foo"); 3132 3133 //auto uriBefore = url; 3134 url = Uri("#anchor"); // everything should remain the same except the anchor 3135 //uriBefore.anchor = "anchor"); 3136 //assert(url == uriBefore); 3137 3138 url = Uri("//example.com"); // same protocol, but different server. the path here should be blank. 3139 3140 url = Uri("//example.com/example.html"); // same protocol, but different server and path 3141 3142 url = Uri("http://example.com/test.html"); // completely absolute link should never be modified 3143 3144 url = Uri("http://example.com"); // completely absolute link should never be modified, even if it has no path 3145 3146 // FIXME: add something for port too 3147 } 3148 3149 // these are like javascript's location.search and location.hash 3150 string search() const { 3151 return query.length ? ("?" ~ query) : ""; 3152 } 3153 string hash() const { 3154 return fragment.length ? ("#" ~ fragment) : ""; 3155 } 3156 } 3157 3158 3159 /* 3160 for session, see web.d 3161 */ 3162 3163 /// breaks down a url encoded string 3164 string[][string] decodeVariables(string data, string separator = "&", string[]* namesInOrder = null, string[]* valuesInOrder = null) { 3165 auto vars = data.split(separator); 3166 string[][string] _get; 3167 foreach(var; vars) { 3168 auto equal = var.indexOf("="); 3169 string name; 3170 string value; 3171 if(equal == -1) { 3172 name = decodeComponent(var); 3173 value = ""; 3174 } else { 3175 //_get[decodeComponent(var[0..equal])] ~= decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3176 // stupid + -> space conversion. 3177 name = decodeComponent(var[0..equal].replace("+", " ")); 3178 value = decodeComponent(var[equal + 1 .. $].replace("+", " ")); 3179 } 3180 3181 _get[name] ~= value; 3182 if(namesInOrder) 3183 (*namesInOrder) ~= name; 3184 if(valuesInOrder) 3185 (*valuesInOrder) ~= value; 3186 } 3187 return _get; 3188 } 3189 3190 /// breaks down a url encoded string, but only returns the last value of any array 3191 string[string] decodeVariablesSingle(string data) { 3192 string[string] va; 3193 auto varArray = decodeVariables(data); 3194 foreach(k, v; varArray) 3195 va[k] = v[$-1]; 3196 3197 return va; 3198 } 3199 3200 /// url encodes the whole string 3201 string encodeVariables(in string[string] data) { 3202 string ret; 3203 3204 bool outputted = false; 3205 foreach(k, v; data) { 3206 if(outputted) 3207 ret ~= "&"; 3208 else 3209 outputted = true; 3210 3211 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3212 } 3213 3214 return ret; 3215 } 3216 3217 /// url encodes a whole string 3218 string encodeVariables(in string[][string] data) { 3219 string ret; 3220 3221 bool outputted = false; 3222 foreach(k, arr; data) { 3223 foreach(v; arr) { 3224 if(outputted) 3225 ret ~= "&"; 3226 else 3227 outputted = true; 3228 ret ~= std.uri.encodeComponent(k) ~ "=" ~ std.uri.encodeComponent(v); 3229 } 3230 } 3231 3232 return ret; 3233 } 3234 3235 /// Encodes all but the explicitly unreserved characters per rfc 3986 3236 /// Alphanumeric and -_.~ are the only ones left unencoded 3237 /// name is borrowed from php 3238 string rawurlencode(in char[] data) { 3239 string ret; 3240 ret.reserve(data.length * 2); 3241 foreach(char c; data) { 3242 if( 3243 (c >= 'a' && c <= 'z') || 3244 (c >= 'A' && c <= 'Z') || 3245 (c >= '0' && c <= '9') || 3246 c == '-' || c == '_' || c == '.' || c == '~') 3247 { 3248 ret ~= c; 3249 } else { 3250 ret ~= '%'; 3251 // since we iterate on char, this should give us the octets of the full utf8 string 3252 ret ~= toHexUpper(c); 3253 } 3254 } 3255 3256 return ret; 3257 } 3258 3259 3260 // http helper functions 3261 3262 // for chunked responses (which embedded http does whenever possible) 3263 version(none) // this is moved up above to avoid making a copy of the data 3264 const(ubyte)[] makeChunk(const(ubyte)[] data) { 3265 const(ubyte)[] ret; 3266 3267 ret = cast(const(ubyte)[]) toHex(data.length); 3268 ret ~= cast(const(ubyte)[]) "\r\n"; 3269 ret ~= data; 3270 ret ~= cast(const(ubyte)[]) "\r\n"; 3271 3272 return ret; 3273 } 3274 3275 string toHex(long num) { 3276 string ret; 3277 while(num) { 3278 int v = num % 16; 3279 num /= 16; 3280 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'a'); 3281 ret ~= d; 3282 } 3283 3284 return to!string(array(ret.retro)); 3285 } 3286 3287 string toHexUpper(long num) { 3288 string ret; 3289 while(num) { 3290 int v = num % 16; 3291 num /= 16; 3292 char d = cast(char) ((v < 10) ? v + '0' : (v-10) + 'A'); 3293 ret ~= d; 3294 } 3295 3296 if(ret.length == 1) 3297 ret ~= "0"; // url encoding requires two digits and that's what this function is used for... 3298 3299 return to!string(array(ret.retro)); 3300 } 3301 3302 3303 // the generic mixins 3304 3305 /++ 3306 Use this instead of writing your own main 3307 3308 It ultimately calls [cgiMainImpl] which creates a [RequestServer] for you. 3309 +/ 3310 mixin template GenericMain(alias fun, long maxContentLength = defaultMaxContentLength) { 3311 mixin CustomCgiMain!(Cgi, fun, maxContentLength); 3312 } 3313 3314 /++ 3315 Boilerplate mixin for a main function that uses the [dispatcher] function. 3316 3317 You can send `typeof(null)` as the `Presenter` argument to use a generic one. 3318 3319 History: 3320 Added July 9, 2021 3321 +/ 3322 mixin template DispatcherMain(Presenter, DispatcherArgs...) { 3323 /++ 3324 Handler to the generated presenter you can use from your objects, etc. 3325 +/ 3326 Presenter activePresenter; 3327 3328 /++ 3329 Request handler that creates the presenter then forwards to the [dispatcher] function. 3330 Renders 404 if the dispatcher did not handle the request. 3331 3332 Will automatically serve the presenter.style and presenter.script as "style.css" and "script.js" 3333 +/ 3334 void handler(Cgi cgi) { 3335 auto presenter = new Presenter; 3336 activePresenter = presenter; 3337 scope(exit) activePresenter = null; 3338 3339 if(cgi.dispatcher!DispatcherArgs(presenter)) 3340 return; 3341 3342 switch(cgi.pathInfo) { 3343 case "/style.css": 3344 cgi.setCache(true); 3345 cgi.setResponseContentType("text/css"); 3346 cgi.write(presenter.style(), true); 3347 break; 3348 case "/script.js": 3349 cgi.setCache(true); 3350 cgi.setResponseContentType("application/javascript"); 3351 cgi.write(presenter.script(), true); 3352 break; 3353 default: 3354 presenter.renderBasicError(cgi, 404); 3355 } 3356 } 3357 mixin GenericMain!handler; 3358 } 3359 3360 mixin template DispatcherMain(DispatcherArgs...) if(!is(DispatcherArgs[0] : WebPresenter!T, T)) { 3361 class GenericPresenter : WebPresenter!GenericPresenter {} 3362 mixin DispatcherMain!(GenericPresenter, DispatcherArgs); 3363 } 3364 3365 private string simpleHtmlEncode(string s) { 3366 return s.replace("&", "&").replace("<", "<").replace(">", ">").replace("\n", "<br />\n"); 3367 } 3368 3369 string messageFromException(Throwable t) { 3370 string message; 3371 if(t !is null) { 3372 debug message = t.toString(); 3373 else message = "An unexpected error has occurred."; 3374 } else { 3375 message = "Unknown error"; 3376 } 3377 return message; 3378 } 3379 3380 string plainHttpError(bool isCgi, string type, Throwable t) { 3381 auto message = messageFromException(t); 3382 message = simpleHtmlEncode(message); 3383 3384 return format("%s %s\r\nContent-Length: %s\r\n\r\n%s", 3385 isCgi ? "Status:" : "HTTP/1.0", 3386 type, message.length, message); 3387 } 3388 3389 // returns true if we were able to recover reasonably 3390 bool handleException(Cgi cgi, Throwable t) { 3391 if(cgi.isClosed) { 3392 // if the channel has been explicitly closed, we can't handle it here 3393 return true; 3394 } 3395 3396 if(cgi.outputtedResponseData) { 3397 // the headers are sent, but the channel is open... since it closes if all was sent, we can append an error message here. 3398 return false; // but I don't want to, since I don't know what condition the output is in; I don't want to inject something (nor check the content-type for that matter. So we say it was not a clean handling. 3399 } else { 3400 // no headers are sent, we can send a full blown error and recover 3401 cgi.setCache(false); 3402 cgi.setResponseContentType("text/html"); 3403 cgi.setResponseLocation(null); // cancel the redirect 3404 cgi.setResponseStatus("500 Internal Server Error"); 3405 cgi.write(simpleHtmlEncode(messageFromException(t))); 3406 cgi.close(); 3407 return true; 3408 } 3409 } 3410 3411 bool isCgiRequestMethod(string s) { 3412 s = s.toUpper(); 3413 if(s == "COMMANDLINE") 3414 return true; 3415 foreach(member; __traits(allMembers, Cgi.RequestMethod)) 3416 if(s == member) 3417 return true; 3418 return false; 3419 } 3420 3421 /// If you want to use a subclass of Cgi with generic main, use this mixin. 3422 mixin template CustomCgiMain(CustomCgi, alias fun, long maxContentLength = defaultMaxContentLength) if(is(CustomCgi : Cgi)) { 3423 // kinda hacky - the T... is passed to Cgi's constructor in standard cgi mode, and ignored elsewhere 3424 void main(string[] args) { 3425 cgiMainImpl!(fun, CustomCgi, maxContentLength)(args); 3426 } 3427 } 3428 3429 version(embedded_httpd_processes) 3430 __gshared int processPoolSize = 8; 3431 3432 // Returns true if run. You should exit the program after that. 3433 bool tryAddonServers(string[] args) { 3434 if(args.length > 1) { 3435 // run the special separate processes if needed 3436 switch(args[1]) { 3437 case "--websocket-server": 3438 version(with_addon_servers) 3439 websocketServers[args[2]](args[3 .. $]); 3440 else 3441 printf("Add-on servers not compiled in.\n"); 3442 return true; 3443 case "--websocket-servers": 3444 import core.demangle; 3445 version(with_addon_servers_connections) 3446 foreach(k, v; websocketServers) 3447 writeln(k, "\t", demangle(k)); 3448 return true; 3449 case "--session-server": 3450 version(with_addon_servers) 3451 runSessionServer(); 3452 else 3453 printf("Add-on servers not compiled in.\n"); 3454 return true; 3455 case "--event-server": 3456 version(with_addon_servers) 3457 runEventServer(); 3458 else 3459 printf("Add-on servers not compiled in.\n"); 3460 return true; 3461 case "--timer-server": 3462 version(with_addon_servers) 3463 runTimerServer(); 3464 else 3465 printf("Add-on servers not compiled in.\n"); 3466 return true; 3467 case "--timed-jobs": 3468 import core.demangle; 3469 version(with_addon_servers_connections) 3470 foreach(k, v; scheduledJobHandlers) 3471 writeln(k, "\t", demangle(k)); 3472 return true; 3473 case "--timed-job": 3474 scheduledJobHandlers[args[2]](args[3 .. $]); 3475 return true; 3476 default: 3477 // intentionally blank - do nothing and carry on to run normally 3478 } 3479 } 3480 return false; 3481 } 3482 3483 /// Tries to simulate a request from the command line. Returns true if it does, false if it didn't find the args. 3484 bool trySimulatedRequest(alias fun, CustomCgi = Cgi)(string[] args) if(is(CustomCgi : Cgi)) { 3485 // we support command line thing for easy testing everywhere 3486 // it needs to be called ./app method uri [other args...] 3487 if(args.length >= 3 && isCgiRequestMethod(args[1])) { 3488 Cgi cgi = new CustomCgi(args); 3489 scope(exit) cgi.dispose(); 3490 fun(cgi); 3491 cgi.close(); 3492 return true; 3493 } 3494 return false; 3495 } 3496 3497 /++ 3498 A server control and configuration struct, as a potential alternative to calling [GenericMain] or [cgiMainImpl]. See the source of [cgiMainImpl] to an example of how you can use it. 3499 3500 History: 3501 Added Sept 26, 2020 (release version 8.5). 3502 +/ 3503 struct RequestServer { 3504 /// 3505 string listeningHost = defaultListeningHost(); 3506 /// 3507 ushort listeningPort = defaultListeningPort(); 3508 3509 /++ 3510 Uses a fork() call, if available, to provide additional crash resiliency and possibly improved performance. On the 3511 other hand, if you fork, you must not assume any memory is shared between requests (you shouldn't be anyway though! But 3512 if you have to, you probably want to set this to false and use an explicit threaded server with [serveEmbeddedHttp]) and 3513 [stop] may not work as well. 3514 3515 History: 3516 Added August 12, 2022 (dub v10.9). Previously, this was only configurable through the `-version=cgi_no_fork` 3517 argument to dmd. That version still defines the value of `cgi_use_fork_default`, used to initialize this, for 3518 compatibility. 3519 +/ 3520 bool useFork = cgi_use_fork_default; 3521 3522 /++ 3523 Determines the number of worker threads to spawn per process, for server modes that use worker threads. 0 will use a 3524 default based on the number of cpus modified by the server mode. 3525 3526 History: 3527 Added August 12, 2022 (dub v10.9) 3528 +/ 3529 int numberOfThreads = 0; 3530 3531 /// 3532 this(string defaultHost, ushort defaultPort) { 3533 this.listeningHost = defaultHost; 3534 this.listeningPort = defaultPort; 3535 } 3536 3537 /// 3538 this(ushort defaultPort) { 3539 listeningPort = defaultPort; 3540 } 3541 3542 /++ 3543 Reads the command line arguments into the values here. 3544 3545 Possible arguments are `--listening-host`, `--listening-port` (or `--port`), `--uid`, and `--gid`. 3546 +/ 3547 void configureFromCommandLine(string[] args) { 3548 bool foundPort = false; 3549 bool foundHost = false; 3550 bool foundUid = false; 3551 bool foundGid = false; 3552 foreach(arg; args) { 3553 if(foundPort) { 3554 listeningPort = to!ushort(arg); 3555 foundPort = false; 3556 } 3557 if(foundHost) { 3558 listeningHost = arg; 3559 foundHost = false; 3560 } 3561 if(foundUid) { 3562 privilegesDropToUid = to!uid_t(arg); 3563 foundUid = false; 3564 } 3565 if(foundGid) { 3566 privilegesDropToGid = to!gid_t(arg); 3567 foundGid = false; 3568 } 3569 if(arg == "--listening-host" || arg == "-h" || arg == "/listening-host") 3570 foundHost = true; 3571 else if(arg == "--port" || arg == "-p" || arg == "/port" || arg == "--listening-port") 3572 foundPort = true; 3573 else if(arg == "--uid") 3574 foundUid = true; 3575 else if(arg == "--gid") 3576 foundGid = true; 3577 } 3578 } 3579 3580 version(Windows) { 3581 private alias uid_t = int; 3582 private alias gid_t = int; 3583 } 3584 3585 /// user (uid) to drop privileges to 3586 /// 0 … do nothing 3587 uid_t privilegesDropToUid = 0; 3588 /// group (gid) to drop privileges to 3589 /// 0 … do nothing 3590 gid_t privilegesDropToGid = 0; 3591 3592 private void dropPrivileges() { 3593 version(Posix) { 3594 import core.sys.posix.unistd; 3595 3596 if (privilegesDropToGid != 0 && setgid(privilegesDropToGid) != 0) 3597 throw new Exception("Dropping privileges via setgid() failed."); 3598 3599 if (privilegesDropToUid != 0 && setuid(privilegesDropToUid) != 0) 3600 throw new Exception("Dropping privileges via setuid() failed."); 3601 } 3602 else { 3603 // FIXME: Windows? 3604 //pragma(msg, "Dropping privileges is not implemented for this platform"); 3605 } 3606 3607 // done, set zero 3608 privilegesDropToGid = 0; 3609 privilegesDropToUid = 0; 3610 } 3611 3612 /++ 3613 Serves a single HTTP request on this thread, with an embedded server, then stops. Designed for cases like embedded oauth responders 3614 3615 History: 3616 Added Oct 10, 2020. 3617 Example: 3618 3619 --- 3620 import arsd.cgi; 3621 void main() { 3622 RequestServer server = RequestServer("127.0.0.1", 6789); 3623 string oauthCode; 3624 string oauthScope; 3625 server.serveHttpOnce!((cgi) { 3626 oauthCode = cgi.request("code"); 3627 oauthScope = cgi.request("scope"); 3628 cgi.write("Thank you, please return to the application."); 3629 }); 3630 // use the code and scope given 3631 } 3632 --- 3633 +/ 3634 void serveHttpOnce(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3635 import std.socket; 3636 3637 bool tcp; 3638 void delegate() cleanup; 3639 auto socket = startListening(listeningHost, listeningPort, tcp, cleanup, 1, &dropPrivileges); 3640 auto connection = socket.accept(); 3641 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(connection); 3642 3643 if(cleanup) 3644 cleanup(); 3645 } 3646 3647 /++ 3648 Starts serving requests according to the current configuration. 3649 +/ 3650 void serve(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3651 version(netman_httpd) { 3652 // Obsolete! 3653 3654 import arsd.httpd; 3655 // what about forwarding the other constructor args? 3656 // this probably needs a whole redoing... 3657 serveHttp!CustomCgi(&fun, listeningPort);//5005); 3658 return; 3659 } else 3660 version(embedded_httpd_processes) { 3661 serveEmbeddedHttpdProcesses!(fun, CustomCgi)(this); 3662 } else 3663 version(embedded_httpd_threads) { 3664 serveEmbeddedHttp!(fun, CustomCgi, maxContentLength)(); 3665 } else 3666 version(scgi) { 3667 serveScgi!(fun, CustomCgi, maxContentLength)(); 3668 } else 3669 version(fastcgi) { 3670 serveFastCgi!(fun, CustomCgi, maxContentLength)(this); 3671 } else 3672 version(stdio_http) { 3673 serveSingleHttpConnectionOnStdio!(fun, CustomCgi, maxContentLength)(); 3674 } else { 3675 //version=plain_cgi; 3676 handleCgiRequest!(fun, CustomCgi, maxContentLength)(); 3677 } 3678 } 3679 3680 /++ 3681 Runs the embedded HTTP thread server specifically, regardless of which build configuration you have. 3682 3683 If you want the forking worker process server, you do need to compile with the embedded_httpd_processes config though. 3684 +/ 3685 void serveEmbeddedHttp(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(ThisFor!fun _this) { 3686 globalStopFlag = false; 3687 static if(__traits(isStaticFunction, fun)) 3688 alias funToUse = fun; 3689 else 3690 void funToUse(CustomCgi cgi) { 3691 static if(__VERSION__ > 2097) 3692 __traits(child, _this, fun)(cgi); 3693 else static assert(0, "Not implemented in your compiler version!"); 3694 } 3695 auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadHttpConnection!(CustomCgi, funToUse), null, useFork, numberOfThreads); 3696 manager.listen(); 3697 } 3698 3699 /++ 3700 Runs the embedded SCGI server specifically, regardless of which build configuration you have. 3701 +/ 3702 void serveScgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3703 globalStopFlag = false; 3704 auto manager = new ListeningConnectionManager(listeningHost, listeningPort, &doThreadScgiConnection!(CustomCgi, fun, maxContentLength), null, useFork, numberOfThreads); 3705 manager.listen(); 3706 } 3707 3708 /++ 3709 Serves a single "connection", but the connection is spoken on stdin and stdout instead of on a socket. 3710 3711 Intended for cases like working from systemd, like discussed here: https://forum.dlang.org/post/avmkfdiitirnrenzljwc@forum.dlang.org 3712 3713 History: 3714 Added May 29, 2021 3715 +/ 3716 void serveSingleHttpConnectionOnStdio(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 3717 doThreadHttpConnectionGuts!(CustomCgi, fun, true)(new FakeSocketForStdin()); 3718 } 3719 3720 /++ 3721 The [stop] function sets a flag that request handlers can (and should) check periodically. If a handler doesn't 3722 respond to this flag, the library will force the issue. This determines when and how the issue will be forced. 3723 +/ 3724 enum ForceStop { 3725 /++ 3726 Stops accepting new requests, but lets ones already in the queue start and complete before exiting. 3727 +/ 3728 afterQueuedRequestsComplete, 3729 /++ 3730 Finishes requests already started their handlers, but drops any others in the queue. Streaming handlers 3731 should cooperate and exit gracefully, but if they don't, it will continue waiting for them. 3732 +/ 3733 afterCurrentRequestsComplete, 3734 /++ 3735 Partial response writes will throw an exception, cancelling any streaming response, but complete 3736 writes will continue to process. Request handlers that respect the stop token will also gracefully cancel. 3737 +/ 3738 cancelStreamingRequestsEarly, 3739 /++ 3740 All writes will throw. 3741 +/ 3742 cancelAllRequestsEarly, 3743 /++ 3744 Use OS facilities to forcibly kill running threads. The server process will be in an undefined state after this call (if this call ever returns). 3745 +/ 3746 forciblyTerminate, 3747 } 3748 3749 version(embedded_httpd_processes) {} else 3750 /++ 3751 Stops serving after the current requests are completed. 3752 3753 Bugs: 3754 Not implemented on version=embedded_httpd_processes, version=fastcgi on any system, or embedded_httpd on Windows (it does work on embedded_httpd_hybrid 3755 on Windows however). Only partially implemented on non-Linux posix systems. 3756 3757 You might also try SIGINT perhaps. 3758 3759 The stopPriority is not yet fully implemented. 3760 +/ 3761 static void stop(ForceStop stopPriority = ForceStop.afterCurrentRequestsComplete) { 3762 globalStopFlag = true; 3763 3764 version(Posix) 3765 if(cancelfd > 0) { 3766 ulong a = 1; 3767 core.sys.posix.unistd.write(cancelfd, &a, a.sizeof); 3768 } 3769 version(Windows) 3770 if(iocp) { 3771 foreach(i; 0 .. 16) // FIXME 3772 PostQueuedCompletionStatus(iocp, 0, cast(ULONG_PTR) null, null); 3773 } 3774 } 3775 } 3776 3777 private alias AliasSeq(T...) = T; 3778 3779 version(with_breaking_cgi_features) 3780 mixin(q{ 3781 template ThisFor(alias t) { 3782 static if(__traits(isStaticFunction, t)) { 3783 alias ThisFor = AliasSeq!(); 3784 } else { 3785 alias ThisFor = __traits(parent, t); 3786 } 3787 } 3788 }); 3789 else 3790 alias ThisFor(alias t) = AliasSeq!(); 3791 3792 private __gshared bool globalStopFlag = false; 3793 3794 version(embedded_httpd_processes) 3795 void serveEmbeddedHttpdProcesses(alias fun, CustomCgi = Cgi)(RequestServer params) { 3796 import core.sys.posix.unistd; 3797 import core.sys.posix.sys.socket; 3798 import core.sys.posix.netinet.in_; 3799 //import std.c.linux.socket; 3800 3801 int sock = socket(AF_INET, SOCK_STREAM, 0); 3802 if(sock == -1) 3803 throw new Exception("socket"); 3804 3805 cloexec(sock); 3806 3807 { 3808 3809 sockaddr_in addr; 3810 addr.sin_family = AF_INET; 3811 addr.sin_port = htons(params.listeningPort); 3812 auto lh = params.listeningHost; 3813 if(lh.length) { 3814 if(inet_pton(AF_INET, lh.toStringz(), &addr.sin_addr.s_addr) != 1) 3815 throw new Exception("bad listening host given, please use an IP address.\nExample: --listening-host 127.0.0.1 means listen only on Localhost.\nExample: --listening-host 0.0.0.0 means listen on all interfaces.\nOr you can pass any other single numeric IPv4 address."); 3816 } else 3817 addr.sin_addr.s_addr = INADDR_ANY; 3818 3819 // HACKISH 3820 int on = 1; 3821 setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, on.sizeof); 3822 // end hack 3823 3824 3825 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 3826 close(sock); 3827 throw new Exception("bind"); 3828 } 3829 3830 // FIXME: if this queue is full, it will just ignore it 3831 // and wait for the client to retransmit it. This is an 3832 // obnoxious timeout condition there. 3833 if(sock.listen(128) == -1) { 3834 close(sock); 3835 throw new Exception("listen"); 3836 } 3837 params.dropPrivileges(); 3838 } 3839 3840 version(embedded_httpd_processes_accept_after_fork) {} else { 3841 int pipeReadFd; 3842 int pipeWriteFd; 3843 3844 { 3845 int[2] pipeFd; 3846 if(socketpair(AF_UNIX, SOCK_DGRAM, 0, pipeFd)) { 3847 import core.stdc.errno; 3848 throw new Exception("pipe failed " ~ to!string(errno)); 3849 } 3850 3851 pipeReadFd = pipeFd[0]; 3852 pipeWriteFd = pipeFd[1]; 3853 } 3854 } 3855 3856 3857 int processCount; 3858 pid_t newPid; 3859 reopen: 3860 while(processCount < processPoolSize) { 3861 newPid = fork(); 3862 if(newPid == 0) { 3863 // start serving on the socket 3864 //ubyte[4096] backingBuffer; 3865 for(;;) { 3866 bool closeConnection; 3867 uint i; 3868 sockaddr addr; 3869 i = addr.sizeof; 3870 version(embedded_httpd_processes_accept_after_fork) { 3871 int s = accept(sock, &addr, &i); 3872 int opt = 1; 3873 import core.sys.posix.netinet.tcp; 3874 // the Cgi class does internal buffering, so disabling this 3875 // helps with latency in many cases... 3876 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 3877 cloexec(s); 3878 } else { 3879 int s; 3880 auto readret = read_fd(pipeReadFd, &s, s.sizeof, &s); 3881 if(readret != s.sizeof) { 3882 import core.stdc.errno; 3883 throw new Exception("pipe read failed " ~ to!string(errno)); 3884 } 3885 3886 //writeln("process ", getpid(), " got socket ", s); 3887 } 3888 3889 try { 3890 3891 if(s == -1) 3892 throw new Exception("accept"); 3893 3894 scope(failure) close(s); 3895 //ubyte[__traits(classInstanceSize, BufferedInputRange)] bufferedRangeContainer; 3896 auto ir = new BufferedInputRange(s); 3897 //auto ir = emplace!BufferedInputRange(bufferedRangeContainer, s, backingBuffer); 3898 3899 while(!ir.empty) { 3900 //ubyte[__traits(classInstanceSize, CustomCgi)] cgiContainer; 3901 3902 Cgi cgi; 3903 try { 3904 cgi = new CustomCgi(ir, &closeConnection); 3905 cgi._outputFileHandle = cast(CgiConnectionHandle) s; 3906 // if we have a single process and the browser tries to leave the connection open while concurrently requesting another, it will block everything an deadlock since there's no other server to accept it. By closing after each request in this situation, it tells the browser to serialize for us. 3907 if(processPoolSize <= 1) 3908 closeConnection = true; 3909 //cgi = emplace!CustomCgi(cgiContainer, ir, &closeConnection); 3910 } catch(Throwable t) { 3911 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 3912 // anyway let's kill the connection 3913 version(CRuntime_Musl) { 3914 // LockingTextWriter fails here 3915 // so working around it 3916 auto estr = t.toString(); 3917 stderr.rawWrite(estr); 3918 stderr.rawWrite("\n"); 3919 } else 3920 stderr.writeln(t.toString()); 3921 sendAll(ir.source, plainHttpError(false, "400 Bad Request", t)); 3922 closeConnection = true; 3923 break; 3924 } 3925 assert(cgi !is null); 3926 scope(exit) 3927 cgi.dispose(); 3928 3929 try { 3930 fun(cgi); 3931 cgi.close(); 3932 if(cgi.websocketMode) 3933 closeConnection = true; 3934 } catch(ConnectionException ce) { 3935 closeConnection = true; 3936 } catch(Throwable t) { 3937 // a processing error can be recovered from 3938 version(CRuntime_Musl) { 3939 // LockingTextWriter fails here 3940 // so working around it 3941 auto estr = t.toString(); 3942 stderr.rawWrite(estr); 3943 } else { 3944 stderr.writeln(t.toString); 3945 } 3946 if(!handleException(cgi, t)) 3947 closeConnection = true; 3948 } 3949 3950 if(closeConnection) { 3951 ir.source.close(); 3952 break; 3953 } else { 3954 if(!ir.empty) 3955 ir.popFront(); // get the next 3956 else if(ir.sourceClosed) { 3957 ir.source.close(); 3958 } 3959 } 3960 } 3961 3962 ir.source.close(); 3963 } catch(Throwable t) { 3964 version(CRuntime_Musl) {} else 3965 debug writeln(t); 3966 // most likely cause is a timeout 3967 } 3968 } 3969 } else if(newPid < 0) { 3970 throw new Exception("fork failed"); 3971 } else { 3972 processCount++; 3973 } 3974 } 3975 3976 // the parent should wait for its children... 3977 if(newPid) { 3978 import core.sys.posix.sys.wait; 3979 3980 version(embedded_httpd_processes_accept_after_fork) {} else { 3981 import core.sys.posix.sys.select; 3982 int[] fdQueue; 3983 while(true) { 3984 // writeln("select call"); 3985 int nfds = pipeWriteFd; 3986 if(sock > pipeWriteFd) 3987 nfds = sock; 3988 nfds += 1; 3989 fd_set read_fds; 3990 fd_set write_fds; 3991 FD_ZERO(&read_fds); 3992 FD_ZERO(&write_fds); 3993 FD_SET(sock, &read_fds); 3994 if(fdQueue.length) 3995 FD_SET(pipeWriteFd, &write_fds); 3996 auto ret = select(nfds, &read_fds, &write_fds, null, null); 3997 if(ret == -1) { 3998 import core.stdc.errno; 3999 if(errno == EINTR) 4000 goto try_wait; 4001 else 4002 throw new Exception("wtf select"); 4003 } 4004 4005 int s = -1; 4006 if(FD_ISSET(sock, &read_fds)) { 4007 uint i; 4008 sockaddr addr; 4009 i = addr.sizeof; 4010 s = accept(sock, &addr, &i); 4011 cloexec(s); 4012 import core.sys.posix.netinet.tcp; 4013 int opt = 1; 4014 setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &opt, opt.sizeof); 4015 } 4016 4017 if(FD_ISSET(pipeWriteFd, &write_fds)) { 4018 if(s == -1 && fdQueue.length) { 4019 s = fdQueue[0]; 4020 fdQueue = fdQueue[1 .. $]; // FIXME reuse buffer 4021 } 4022 write_fd(pipeWriteFd, &s, s.sizeof, s); 4023 close(s); // we are done with it, let the other process take ownership 4024 } else 4025 fdQueue ~= s; 4026 } 4027 } 4028 4029 try_wait: 4030 4031 int status; 4032 while(-1 != wait(&status)) { 4033 version(CRuntime_Musl) {} else { 4034 import std.stdio; writeln("Process died ", status); 4035 } 4036 processCount--; 4037 goto reopen; 4038 } 4039 close(sock); 4040 } 4041 } 4042 4043 version(fastcgi) 4044 void serveFastCgi(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(RequestServer params) { 4045 // SetHandler fcgid-script 4046 FCGX_Stream* input, output, error; 4047 FCGX_ParamArray env; 4048 4049 4050 4051 const(ubyte)[] getFcgiChunk() { 4052 const(ubyte)[] ret; 4053 while(FCGX_HasSeenEOF(input) != -1) 4054 ret ~= cast(ubyte) FCGX_GetChar(input); 4055 return ret; 4056 } 4057 4058 void writeFcgi(const(ubyte)[] data) { 4059 FCGX_PutStr(data.ptr, data.length, output); 4060 } 4061 4062 void doARequest() { 4063 string[string] fcgienv; 4064 4065 for(auto e = env; e !is null && *e !is null; e++) { 4066 string cur = to!string(*e); 4067 auto idx = cur.indexOf("="); 4068 string name, value; 4069 if(idx == -1) 4070 name = cur; 4071 else { 4072 name = cur[0 .. idx]; 4073 value = cur[idx + 1 .. $]; 4074 } 4075 4076 fcgienv[name] = value; 4077 } 4078 4079 void flushFcgi() { 4080 FCGX_FFlush(output); 4081 } 4082 4083 Cgi cgi; 4084 try { 4085 cgi = new CustomCgi(maxContentLength, fcgienv, &getFcgiChunk, &writeFcgi, &flushFcgi); 4086 } catch(Throwable t) { 4087 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4088 writeFcgi(cast(const(ubyte)[]) plainHttpError(true, "400 Bad Request", t)); 4089 return; //continue; 4090 } 4091 assert(cgi !is null); 4092 scope(exit) cgi.dispose(); 4093 try { 4094 fun(cgi); 4095 cgi.close(); 4096 } catch(Throwable t) { 4097 // log it to the error stream 4098 FCGX_PutStr(cast(ubyte*) t.msg.ptr, t.msg.length, error); 4099 // handle it for the user, if we can 4100 if(!handleException(cgi, t)) 4101 return; // continue; 4102 } 4103 } 4104 4105 auto lp = params.listeningPort; 4106 auto host = params.listeningHost; 4107 4108 FCGX_Request request; 4109 if(lp || !host.empty) { 4110 // if a listening port was specified on the command line, we want to spawn ourself 4111 // (needed for nginx without spawn-fcgi, e.g. on Windows) 4112 FCGX_Init(); 4113 4114 int sock; 4115 4116 if(host.startsWith("unix:")) { 4117 sock = FCGX_OpenSocket(toStringz(params.listeningHost["unix:".length .. $]), 12); 4118 } else if(host.startsWith("abstract:")) { 4119 sock = FCGX_OpenSocket(toStringz("\0" ~ params.listeningHost["abstract:".length .. $]), 12); 4120 } else { 4121 sock = FCGX_OpenSocket(toStringz(params.listeningHost ~ ":" ~ to!string(lp)), 12); 4122 } 4123 4124 if(sock < 0) 4125 throw new Exception("Couldn't listen on the port"); 4126 FCGX_InitRequest(&request, sock, 0); 4127 while(FCGX_Accept_r(&request) >= 0) { 4128 input = request.inStream; 4129 output = request.outStream; 4130 error = request.errStream; 4131 env = request.envp; 4132 doARequest(); 4133 } 4134 } else { 4135 // otherwise, assume the httpd is doing it (the case for Apache, IIS, and Lighttpd) 4136 // using the version with a global variable since we are separate processes anyway 4137 while(FCGX_Accept(&input, &output, &error, &env) >= 0) { 4138 doARequest(); 4139 } 4140 } 4141 } 4142 4143 /// Returns the default listening port for the current cgi configuration. 8085 for embedded httpd, 4000 for scgi, irrelevant for others. 4144 ushort defaultListeningPort() { 4145 version(netman_httpd) 4146 return 8080; 4147 else version(embedded_httpd_processes) 4148 return 8085; 4149 else version(embedded_httpd_threads) 4150 return 8085; 4151 else version(scgi) 4152 return 4000; 4153 else 4154 return 0; 4155 } 4156 4157 /// Default host for listening. 127.0.0.1 for scgi, null (aka all interfaces) for all others. If you want the server directly accessible from other computers on the network, normally use null. If not, 127.0.0.1 is a bit better. Settable with default handlers with --listening-host command line argument. 4158 string defaultListeningHost() { 4159 version(netman_httpd) 4160 return null; 4161 else version(embedded_httpd_processes) 4162 return null; 4163 else version(embedded_httpd_threads) 4164 return null; 4165 else version(scgi) 4166 return "127.0.0.1"; 4167 else 4168 return null; 4169 4170 } 4171 4172 /++ 4173 This is the function [GenericMain] calls. View its source for some simple boilerplate you can copy/paste and modify, or you can call it yourself from your `main`. 4174 4175 Params: 4176 fun = Your request handler 4177 CustomCgi = a subclass of Cgi, if you wise to customize it further 4178 maxContentLength = max POST size you want to allow 4179 args = command-line arguments 4180 4181 History: 4182 Documented Sept 26, 2020. 4183 +/ 4184 void cgiMainImpl(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)(string[] args) if(is(CustomCgi : Cgi)) { 4185 if(tryAddonServers(args)) 4186 return; 4187 4188 if(trySimulatedRequest!(fun, CustomCgi)(args)) 4189 return; 4190 4191 RequestServer server; 4192 // you can change the port here if you like 4193 // server.listeningPort = 9000; 4194 4195 // then call this to let the command line args override your default 4196 server.configureFromCommandLine(args); 4197 4198 // and serve the request(s). 4199 server.serve!(fun, CustomCgi, maxContentLength)(); 4200 } 4201 4202 //version(plain_cgi) 4203 void handleCgiRequest(alias fun, CustomCgi = Cgi, long maxContentLength = defaultMaxContentLength)() { 4204 // standard CGI is the default version 4205 4206 4207 // Set stdin to binary mode if necessary to avoid mangled newlines 4208 // the fact that stdin is global means this could be trouble but standard cgi request 4209 // handling is one per process anyway so it shouldn't actually be threaded here or anything. 4210 version(Windows) { 4211 version(Win64) 4212 _setmode(std.stdio.stdin.fileno(), 0x8000); 4213 else 4214 setmode(std.stdio.stdin.fileno(), 0x8000); 4215 } 4216 4217 Cgi cgi; 4218 try { 4219 cgi = new CustomCgi(maxContentLength); 4220 version(Posix) 4221 cgi._outputFileHandle = cast(CgiConnectionHandle) 1; // stdout 4222 else version(Windows) 4223 cgi._outputFileHandle = cast(CgiConnectionHandle) GetStdHandle(STD_OUTPUT_HANDLE); 4224 else static assert(0); 4225 } catch(Throwable t) { 4226 version(CRuntime_Musl) { 4227 // LockingTextWriter fails here 4228 // so working around it 4229 auto s = t.toString(); 4230 stderr.rawWrite(s); 4231 stdout.rawWrite(plainHttpError(true, "400 Bad Request", t)); 4232 } else { 4233 stderr.writeln(t.msg); 4234 // the real http server will probably handle this; 4235 // most likely, this is a bug in Cgi. But, oh well. 4236 stdout.write(plainHttpError(true, "400 Bad Request", t)); 4237 } 4238 return; 4239 } 4240 assert(cgi !is null); 4241 scope(exit) cgi.dispose(); 4242 4243 try { 4244 fun(cgi); 4245 cgi.close(); 4246 } catch (Throwable t) { 4247 version(CRuntime_Musl) { 4248 // LockingTextWriter fails here 4249 // so working around it 4250 auto s = t.msg; 4251 stderr.rawWrite(s); 4252 } else { 4253 stderr.writeln(t.msg); 4254 } 4255 if(!handleException(cgi, t)) 4256 return; 4257 } 4258 } 4259 4260 private __gshared int cancelfd = -1; 4261 4262 /+ 4263 The event loop for embedded_httpd_threads will prolly fiber dispatch 4264 cgi constructors too, so slow posts will not monopolize a worker thread. 4265 4266 May want to provide the worker task system just need to ensure all the fibers 4267 has a big enough stack for real work... would also ideally like to reuse them. 4268 4269 4270 So prolly bir would switch it to nonblocking. If it would block, it epoll 4271 registers one shot with this existing fiber to take it over. 4272 4273 new connection comes in. it picks a fiber off the free list, 4274 or if there is none, it creates a new one. this fiber handles 4275 this connection the whole time. 4276 4277 epoll triggers the fiber when something comes in. it is called by 4278 a random worker thread, it might change at any time. at least during 4279 the constructor. maybe into the main body it will stay tied to a thread 4280 just so TLS stuff doesn't randomly change in the middle. but I could 4281 specify if you yield all bets are off. 4282 4283 when the request is finished, if there's more data buffered, it just 4284 keeps going. if there is no more data buffered, it epoll ctls to 4285 get triggered when more data comes in. all one shot. 4286 4287 when a connection is closed, the fiber returns and is then reset 4288 and added to the free list. if the free list is full, the fiber is 4289 just freed, this means it will balloon to a certain size but not generally 4290 grow beyond that unless the activity keeps going. 4291 4292 256 KB stack i thnk per fiber. 4,000 active fibers per gigabyte of memory. 4293 4294 So the fiber has its own magic methods to read and write. if they would block, it registers 4295 for epoll and yields. when it returns, it read/writes and then returns back normal control. 4296 4297 basically you issue the command and it tells you when it is done 4298 4299 it needs to DEL the epoll thing when it is closed. add it when opened. mod it when anther thing issued 4300 4301 +/ 4302 4303 /++ 4304 The stack size when a fiber is created. You can set this from your main or from a shared static constructor 4305 to optimize your memory use if you know you don't need this much space. Be careful though, some functions use 4306 more stack space than you realize and a recursive function (including ones like in dom.d) can easily grow fast! 4307 4308 History: 4309 Added July 10, 2021. Previously, it used the druntime default of 16 KB. 4310 +/ 4311 version(cgi_use_fiber) 4312 __gshared size_t fiberStackSize = 4096 * 100; 4313 4314 version(cgi_use_fiber) 4315 class CgiFiber : Fiber { 4316 private void function(Socket) f_handler; 4317 private void f_handler_dg(Socket s) { // to avoid extra allocation w/ function 4318 f_handler(s); 4319 } 4320 this(void function(Socket) handler) { 4321 this.f_handler = handler; 4322 this(&f_handler_dg); 4323 } 4324 4325 this(void delegate(Socket) handler) { 4326 this.handler = handler; 4327 super(&run, fiberStackSize); 4328 } 4329 4330 Socket connection; 4331 void delegate(Socket) handler; 4332 4333 void run() { 4334 handler(connection); 4335 } 4336 4337 void delegate() postYield; 4338 4339 private void setPostYield(scope void delegate() py) @nogc { 4340 postYield = cast(void delegate()) py; 4341 } 4342 4343 void proceed() { 4344 try { 4345 call(); 4346 auto py = postYield; 4347 postYield = null; 4348 if(py !is null) 4349 py(); 4350 } catch(Exception e) { 4351 if(connection) 4352 connection.close(); 4353 goto terminate; 4354 } 4355 4356 if(state == State.TERM) { 4357 terminate: 4358 import core.memory; 4359 GC.removeRoot(cast(void*) this); 4360 } 4361 } 4362 } 4363 4364 version(cgi_use_fiber) 4365 version(Windows) { 4366 4367 extern(Windows) private { 4368 4369 import core.sys.windows.mswsock; 4370 4371 alias GROUP=uint; 4372 alias LPWSAPROTOCOL_INFOW = void*; 4373 SOCKET WSASocketW(int af, int type, int protocol, LPWSAPROTOCOL_INFOW lpProtocolInfo, GROUP g, DWORD dwFlags); 4374 int WSASend(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesSent, DWORD dwFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4375 int WSARecv(SOCKET s, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags, LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine); 4376 4377 struct WSABUF { 4378 ULONG len; 4379 CHAR *buf; 4380 } 4381 alias LPWSABUF = WSABUF*; 4382 4383 alias WSAOVERLAPPED = OVERLAPPED; 4384 alias LPWSAOVERLAPPED = LPOVERLAPPED; 4385 /+ 4386 4387 alias LPFN_ACCEPTEX = 4388 BOOL 4389 function( 4390 SOCKET sListenSocket, 4391 SOCKET sAcceptSocket, 4392 //_Out_writes_bytes_(dwReceiveDataLength+dwLocalAddressLength+dwRemoteAddressLength) PVOID lpOutputBuffer, 4393 void* lpOutputBuffer, 4394 WORD dwReceiveDataLength, 4395 WORD dwLocalAddressLength, 4396 WORD dwRemoteAddressLength, 4397 LPDWORD lpdwBytesReceived, 4398 LPOVERLAPPED lpOverlapped 4399 ); 4400 4401 enum WSAID_ACCEPTEX = GUID([0xb5367df1,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]]); 4402 +/ 4403 4404 enum WSAID_GETACCEPTEXSOCKADDRS = GUID(0xb5367df2,0xcbac,0x11cf,[0x95,0xca,0x00,0x80,0x5f,0x48,0xa1,0x92]); 4405 } 4406 4407 private class PseudoblockingOverlappedSocket : Socket { 4408 SOCKET handle; 4409 4410 CgiFiber fiber; 4411 4412 this(AddressFamily af, SocketType st) { 4413 auto handle = WSASocketW(af, st, 0, null, 0, 1 /*WSA_FLAG_OVERLAPPED*/); 4414 if(!handle) 4415 throw new Exception("WSASocketW"); 4416 this.handle = handle; 4417 4418 iocp = CreateIoCompletionPort(cast(HANDLE) handle, iocp, cast(ULONG_PTR) cast(void*) this, 0); 4419 4420 if(iocp is null) { 4421 writeln(GetLastError()); 4422 throw new Exception("CreateIoCompletionPort"); 4423 } 4424 4425 super(cast(socket_t) handle, af); 4426 } 4427 this() pure nothrow @trusted { assert(0); } 4428 4429 override void blocking(bool) {} // meaningless to us, just ignore it. 4430 4431 protected override Socket accepting() pure nothrow { 4432 assert(0); 4433 } 4434 4435 bool addressesParsed; 4436 Address la; 4437 Address ra; 4438 4439 private void populateAddresses() { 4440 if(addressesParsed) 4441 return; 4442 addressesParsed = true; 4443 4444 int lalen, ralen; 4445 4446 sockaddr_in* la; 4447 sockaddr_in* ra; 4448 4449 lpfnGetAcceptExSockaddrs( 4450 scratchBuffer.ptr, 4451 0, // same as in the AcceptEx call! 4452 sockaddr_in.sizeof + 16, 4453 sockaddr_in.sizeof + 16, 4454 cast(sockaddr**) &la, 4455 &lalen, 4456 cast(sockaddr**) &ra, 4457 &ralen 4458 ); 4459 4460 if(la) 4461 this.la = new InternetAddress(*la); 4462 if(ra) 4463 this.ra = new InternetAddress(*ra); 4464 4465 } 4466 4467 override @property @trusted Address localAddress() { 4468 populateAddresses(); 4469 return la; 4470 } 4471 override @property @trusted Address remoteAddress() { 4472 populateAddresses(); 4473 return ra; 4474 } 4475 4476 PseudoblockingOverlappedSocket accepted; 4477 4478 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4479 __gshared static typeof(&GetAcceptExSockaddrs) lpfnGetAcceptExSockaddrs; 4480 4481 override Socket accept() @trusted { 4482 __gshared static LPFN_ACCEPTEX lpfnAcceptEx; 4483 4484 if(lpfnAcceptEx is null) { 4485 DWORD dwBytes; 4486 GUID GuidAcceptEx = WSAID_ACCEPTEX; 4487 4488 auto iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4489 &GuidAcceptEx, GuidAcceptEx.sizeof, 4490 &lpfnAcceptEx, lpfnAcceptEx.sizeof, 4491 &dwBytes, null, null); 4492 4493 GuidAcceptEx = WSAID_GETACCEPTEXSOCKADDRS; 4494 iResult = WSAIoctl(handle, 0xc8000006 /*SIO_GET_EXTENSION_FUNCTION_POINTER*/, 4495 &GuidAcceptEx, GuidAcceptEx.sizeof, 4496 &lpfnGetAcceptExSockaddrs, lpfnGetAcceptExSockaddrs.sizeof, 4497 &dwBytes, null, null); 4498 4499 } 4500 4501 auto pfa = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 4502 accepted = pfa; 4503 4504 SOCKET pendingForAccept = pfa.handle; 4505 DWORD ignored; 4506 4507 auto ret = lpfnAcceptEx(handle, 4508 pendingForAccept, 4509 // buffer to receive up front 4510 pfa.scratchBuffer.ptr, 4511 0, 4512 // size of local and remote addresses. normally + 16. 4513 sockaddr_in.sizeof + 16, 4514 sockaddr_in.sizeof + 16, 4515 &ignored, // bytes would be given through the iocp instead but im not even requesting the thing 4516 &overlapped 4517 ); 4518 4519 return pfa; 4520 } 4521 4522 override void connect(Address to) { assert(0); } 4523 4524 DWORD lastAnswer; 4525 ubyte[1024] scratchBuffer; 4526 static assert(scratchBuffer.length > sockaddr_in.sizeof * 2 + 32); 4527 4528 WSABUF[1] buffer; 4529 OVERLAPPED overlapped; 4530 override ptrdiff_t send(scope const(void)[] buf, SocketFlags flags) @trusted { 4531 overlapped = overlapped.init; 4532 buffer[0].len = cast(DWORD) buf.length; 4533 buffer[0].buf = cast(CHAR*) buf.ptr; 4534 fiber.setPostYield( () { 4535 if(!WSASend(handle, buffer.ptr, cast(DWORD) buffer.length, null, 0, &overlapped, null)) { 4536 if(GetLastError() != 997) { 4537 //throw new Exception("WSASend fail"); 4538 } 4539 } 4540 }); 4541 4542 Fiber.yield(); 4543 return lastAnswer; 4544 } 4545 override ptrdiff_t receive(scope void[] buf, SocketFlags flags) @trusted { 4546 overlapped = overlapped.init; 4547 buffer[0].len = cast(DWORD) buf.length; 4548 buffer[0].buf = cast(CHAR*) buf.ptr; 4549 4550 DWORD flags2 = 0; 4551 4552 fiber.setPostYield(() { 4553 if(!WSARecv(handle, buffer.ptr, cast(DWORD) buffer.length, null, &flags2 /* flags */, &overlapped, null)) { 4554 if(GetLastError() != 997) { 4555 //writeln("WSARecv ", WSAGetLastError()); 4556 //throw new Exception("WSARecv fail"); 4557 } 4558 } 4559 }); 4560 4561 Fiber.yield(); 4562 return lastAnswer; 4563 } 4564 4565 // I might go back and implement these for udp things. 4566 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags, ref Address from) @trusted { 4567 assert(0); 4568 } 4569 override ptrdiff_t receiveFrom(scope void[] buf, SocketFlags flags) @trusted { 4570 assert(0); 4571 } 4572 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags, Address to) @trusted { 4573 assert(0); 4574 } 4575 override ptrdiff_t sendTo(scope const(void)[] buf, SocketFlags flags) @trusted { 4576 assert(0); 4577 } 4578 4579 // lol overload sets 4580 alias send = typeof(super).send; 4581 alias receive = typeof(super).receive; 4582 alias sendTo = typeof(super).sendTo; 4583 alias receiveFrom = typeof(super).receiveFrom; 4584 4585 } 4586 } 4587 4588 void doThreadHttpConnection(CustomCgi, alias fun)(Socket connection) { 4589 assert(connection !is null); 4590 version(cgi_use_fiber) { 4591 auto fiber = new CgiFiber(&doThreadHttpConnectionGuts!(CustomCgi, fun)); 4592 4593 version(Windows) { 4594 (cast(PseudoblockingOverlappedSocket) connection).fiber = fiber; 4595 } 4596 4597 import core.memory; 4598 GC.addRoot(cast(void*) fiber); 4599 fiber.connection = connection; 4600 fiber.proceed(); 4601 } else { 4602 doThreadHttpConnectionGuts!(CustomCgi, fun)(connection); 4603 } 4604 } 4605 4606 void doThreadHttpConnectionGuts(CustomCgi, alias fun, bool alwaysCloseConnection = false)(Socket connection) { 4607 scope(failure) { 4608 // catch all for other errors 4609 try { 4610 sendAll(connection, plainHttpError(false, "500 Internal Server Error", null)); 4611 connection.close(); 4612 } catch(Exception e) {} // swallow it, we're aborting anyway. 4613 } 4614 4615 bool closeConnection = alwaysCloseConnection; 4616 4617 /+ 4618 ubyte[4096] inputBuffer = void; 4619 ubyte[__traits(classInstanceSize, BufferedInputRange)] birBuffer = void; 4620 ubyte[__traits(classInstanceSize, CustomCgi)] cgiBuffer = void; 4621 4622 birBuffer[] = cast(ubyte[]) typeid(BufferedInputRange).initializer()[]; 4623 BufferedInputRange ir = cast(BufferedInputRange) cast(void*) birBuffer.ptr; 4624 ir.__ctor(connection, inputBuffer[], true); 4625 +/ 4626 4627 auto ir = new BufferedInputRange(connection); 4628 4629 while(!ir.empty) { 4630 4631 if(ir.view.length == 0) { 4632 ir.popFront(); 4633 if(ir.sourceClosed) { 4634 connection.close(); 4635 closeConnection = true; 4636 break; 4637 } 4638 } 4639 4640 Cgi cgi; 4641 try { 4642 cgi = new CustomCgi(ir, &closeConnection); 4643 // There's a bunch of these casts around because the type matches up with 4644 // the -version=.... specifiers, just you can also create a RequestServer 4645 // and instantiate the things where the types don't match up. It isn't exactly 4646 // correct but I also don't care rn. Might FIXME and either remove it later or something. 4647 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 4648 } catch(ConnectionClosedException ce) { 4649 closeConnection = true; 4650 break; 4651 } catch(ConnectionException ce) { 4652 // broken pipe or something, just abort the connection 4653 closeConnection = true; 4654 break; 4655 } catch(Throwable t) { 4656 // a construction error is either bad code or bad request; bad request is what it should be since this is bug free :P 4657 // anyway let's kill the connection 4658 version(CRuntime_Musl) { 4659 stderr.rawWrite(t.toString()); 4660 stderr.rawWrite("\n"); 4661 } else { 4662 stderr.writeln(t.toString()); 4663 } 4664 sendAll(connection, plainHttpError(false, "400 Bad Request", t)); 4665 closeConnection = true; 4666 break; 4667 } 4668 assert(cgi !is null); 4669 scope(exit) 4670 cgi.dispose(); 4671 4672 try { 4673 fun(cgi); 4674 cgi.close(); 4675 if(cgi.websocketMode) 4676 closeConnection = true; 4677 } catch(ConnectionException ce) { 4678 // broken pipe or something, just abort the connection 4679 closeConnection = true; 4680 } catch(ConnectionClosedException ce) { 4681 // broken pipe or something, just abort the connection 4682 closeConnection = true; 4683 } catch(Throwable t) { 4684 // a processing error can be recovered from 4685 version(CRuntime_Musl) {} else 4686 stderr.writeln(t.toString); 4687 if(!handleException(cgi, t)) 4688 closeConnection = true; 4689 } 4690 4691 if(globalStopFlag) 4692 closeConnection = true; 4693 4694 if(closeConnection || alwaysCloseConnection) { 4695 connection.shutdown(SocketShutdown.BOTH); 4696 connection.close(); 4697 ir.dispose(); 4698 closeConnection = false; // don't reclose after loop 4699 break; 4700 } else { 4701 if(ir.front.length) { 4702 ir.popFront(); // we can't just discard the buffer, so get the next bit and keep chugging along 4703 } else if(ir.sourceClosed) { 4704 ir.source.shutdown(SocketShutdown.BOTH); 4705 ir.source.close(); 4706 ir.dispose(); 4707 closeConnection = false; 4708 } else { 4709 continue; 4710 // break; // this was for a keepalive experiment 4711 } 4712 } 4713 } 4714 4715 if(closeConnection) { 4716 connection.shutdown(SocketShutdown.BOTH); 4717 connection.close(); 4718 ir.dispose(); 4719 } 4720 4721 // I am otherwise NOT closing it here because the parent thread might still be able to make use of the keep-alive connection! 4722 } 4723 4724 void doThreadScgiConnection(CustomCgi, alias fun, long maxContentLength)(Socket connection) { 4725 // and now we can buffer 4726 scope(failure) 4727 connection.close(); 4728 4729 import al = std.algorithm; 4730 4731 size_t size; 4732 4733 string[string] headers; 4734 4735 auto range = new BufferedInputRange(connection); 4736 more_data: 4737 auto chunk = range.front(); 4738 // waiting for colon for header length 4739 auto idx = indexOf(cast(string) chunk, ':'); 4740 if(idx == -1) { 4741 try { 4742 range.popFront(); 4743 } catch(Exception e) { 4744 // it is just closed, no big deal 4745 connection.close(); 4746 return; 4747 } 4748 goto more_data; 4749 } 4750 4751 size = to!size_t(cast(string) chunk[0 .. idx]); 4752 chunk = range.consume(idx + 1); 4753 // reading headers 4754 if(chunk.length < size) 4755 range.popFront(0, size + 1); 4756 // we are now guaranteed to have enough 4757 chunk = range.front(); 4758 assert(chunk.length > size); 4759 4760 idx = 0; 4761 string key; 4762 string value; 4763 foreach(part; al.splitter(chunk, '\0')) { 4764 if(idx & 1) { // odd is value 4765 value = cast(string)(part.idup); 4766 headers[key] = value; // commit 4767 } else 4768 key = cast(string)(part.idup); 4769 idx++; 4770 } 4771 4772 enforce(chunk[size] == ','); // the terminator 4773 4774 range.consume(size + 1); 4775 // reading data 4776 // this will be done by Cgi 4777 4778 const(ubyte)[] getScgiChunk() { 4779 // we are already primed 4780 auto data = range.front(); 4781 if(data.length == 0 && !range.sourceClosed) { 4782 range.popFront(0); 4783 data = range.front(); 4784 } else if (range.sourceClosed) 4785 range.source.close(); 4786 4787 return data; 4788 } 4789 4790 void writeScgi(const(ubyte)[] data) { 4791 sendAll(connection, data); 4792 } 4793 4794 void flushScgi() { 4795 // I don't *think* I have to do anything.... 4796 } 4797 4798 Cgi cgi; 4799 try { 4800 cgi = new CustomCgi(maxContentLength, headers, &getScgiChunk, &writeScgi, &flushScgi); 4801 cgi._outputFileHandle = cast(CgiConnectionHandle) connection.handle; 4802 } catch(Throwable t) { 4803 sendAll(connection, plainHttpError(true, "400 Bad Request", t)); 4804 connection.close(); 4805 return; // this connection is dead 4806 } 4807 assert(cgi !is null); 4808 scope(exit) cgi.dispose(); 4809 try { 4810 fun(cgi); 4811 cgi.close(); 4812 connection.close(); 4813 } catch(Throwable t) { 4814 // no std err 4815 if(!handleException(cgi, t)) { 4816 connection.close(); 4817 return; 4818 } else { 4819 connection.close(); 4820 return; 4821 } 4822 } 4823 } 4824 4825 string printDate(DateTime date) { 4826 char[29] buffer = void; 4827 printDateToBuffer(date, buffer[]); 4828 return buffer.idup; 4829 } 4830 4831 int printDateToBuffer(DateTime date, char[] buffer) @nogc { 4832 assert(buffer.length >= 29); 4833 // 29 static length ? 4834 4835 static immutable daysOfWeek = [ 4836 "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" 4837 ]; 4838 4839 static immutable months = [ 4840 null, "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" 4841 ]; 4842 4843 buffer[0 .. 3] = daysOfWeek[date.dayOfWeek]; 4844 buffer[3 .. 5] = ", "; 4845 buffer[5] = date.day / 10 + '0'; 4846 buffer[6] = date.day % 10 + '0'; 4847 buffer[7] = ' '; 4848 buffer[8 .. 11] = months[date.month]; 4849 buffer[11] = ' '; 4850 auto y = date.year; 4851 buffer[12] = cast(char) (y / 1000 + '0'); y %= 1000; 4852 buffer[13] = cast(char) (y / 100 + '0'); y %= 100; 4853 buffer[14] = cast(char) (y / 10 + '0'); y %= 10; 4854 buffer[15] = cast(char) (y + '0'); 4855 buffer[16] = ' '; 4856 buffer[17] = date.hour / 10 + '0'; 4857 buffer[18] = date.hour % 10 + '0'; 4858 buffer[19] = ':'; 4859 buffer[20] = date.minute / 10 + '0'; 4860 buffer[21] = date.minute % 10 + '0'; 4861 buffer[22] = ':'; 4862 buffer[23] = date.second / 10 + '0'; 4863 buffer[24] = date.second % 10 + '0'; 4864 buffer[25 .. $] = " GMT"; 4865 4866 return 29; 4867 } 4868 4869 4870 // Referencing this gigantic typeid seems to remind the compiler 4871 // to actually put the symbol in the object file. I guess the immutable 4872 // assoc array array isn't actually included in druntime 4873 void hackAroundLinkerError() { 4874 stdout.rawWrite(typeid(const(immutable(char)[][])[immutable(char)[]]).toString()); 4875 stdout.rawWrite(typeid(immutable(char)[][][immutable(char)[]]).toString()); 4876 stdout.rawWrite(typeid(Cgi.UploadedFile[immutable(char)[]]).toString()); 4877 stdout.rawWrite(typeid(Cgi.UploadedFile[][immutable(char)[]]).toString()); 4878 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile)[immutable(char)[]]).toString()); 4879 stdout.rawWrite(typeid(immutable(Cgi.UploadedFile[])[immutable(char)[]]).toString()); 4880 stdout.rawWrite(typeid(immutable(char[])[immutable(char)[]]).toString()); 4881 // this is getting kinda ridiculous btw. Moving assoc arrays 4882 // to the library is the pain that keeps on coming. 4883 4884 // eh this broke the build on the work server 4885 // stdout.rawWrite(typeid(immutable(char)[][immutable(string[])])); 4886 stdout.rawWrite(typeid(immutable(string[])[immutable(char)[]]).toString()); 4887 } 4888 4889 4890 4891 4892 4893 version(fastcgi) { 4894 pragma(lib, "fcgi"); 4895 4896 static if(size_t.sizeof == 8) // 64 bit 4897 alias long c_int; 4898 else 4899 alias int c_int; 4900 4901 extern(C) { 4902 struct FCGX_Stream { 4903 ubyte* rdNext; 4904 ubyte* wrNext; 4905 ubyte* stop; 4906 ubyte* stopUnget; 4907 c_int isReader; 4908 c_int isClosed; 4909 c_int wasFCloseCalled; 4910 c_int FCGI_errno; 4911 void* function(FCGX_Stream* stream) fillBuffProc; 4912 void* function(FCGX_Stream* stream, c_int doClose) emptyBuffProc; 4913 void* data; 4914 } 4915 4916 // note: this is meant to be opaque, so don't access it directly 4917 struct FCGX_Request { 4918 int requestId; 4919 int role; 4920 FCGX_Stream* inStream; 4921 FCGX_Stream* outStream; 4922 FCGX_Stream* errStream; 4923 char** envp; 4924 void* paramsPtr; 4925 int ipcFd; 4926 int isBeginProcessed; 4927 int keepConnection; 4928 int appStatus; 4929 int nWriters; 4930 int flags; 4931 int listen_sock; 4932 } 4933 4934 int FCGX_InitRequest(FCGX_Request *request, int sock, int flags); 4935 void FCGX_Init(); 4936 4937 int FCGX_Accept_r(FCGX_Request *request); 4938 4939 4940 alias char** FCGX_ParamArray; 4941 4942 c_int FCGX_Accept(FCGX_Stream** stdin, FCGX_Stream** stdout, FCGX_Stream** stderr, FCGX_ParamArray* envp); 4943 c_int FCGX_GetChar(FCGX_Stream* stream); 4944 c_int FCGX_PutStr(const ubyte* str, c_int n, FCGX_Stream* stream); 4945 int FCGX_HasSeenEOF(FCGX_Stream* stream); 4946 c_int FCGX_FFlush(FCGX_Stream *stream); 4947 4948 int FCGX_OpenSocket(in char*, int); 4949 } 4950 } 4951 4952 4953 /* This might go int a separate module eventually. It is a network input helper class. */ 4954 4955 import std.socket; 4956 4957 version(cgi_use_fiber) { 4958 import core.thread; 4959 4960 version(linux) { 4961 import core.sys.linux.epoll; 4962 4963 int epfd = -1; // thread local because EPOLLEXCLUSIVE works much better this way... weirdly. 4964 } else version(Windows) { 4965 // declaring the iocp thing below... 4966 } else static assert(0, "The hybrid fiber server is not implemented on your OS."); 4967 } 4968 4969 version(Windows) 4970 __gshared HANDLE iocp; 4971 4972 version(cgi_use_fiber) { 4973 version(linux) 4974 private enum WakeupEvent { 4975 Read = EPOLLIN, 4976 Write = EPOLLOUT 4977 } 4978 else version(Windows) 4979 private enum WakeupEvent { 4980 Read, Write 4981 } 4982 else static assert(0); 4983 } 4984 4985 version(cgi_use_fiber) 4986 private void registerEventWakeup(bool* registered, Socket source, WakeupEvent e) @nogc { 4987 4988 // static cast since I know what i have in here and don't want to pay for dynamic cast 4989 auto f = cast(CgiFiber) cast(void*) Fiber.getThis(); 4990 4991 version(linux) { 4992 f.setPostYield = () { 4993 if(*registered) { 4994 // rearm 4995 epoll_event evt; 4996 evt.events = e | EPOLLONESHOT; 4997 evt.data.ptr = cast(void*) f; 4998 if(epoll_ctl(epfd, EPOLL_CTL_MOD, source.handle, &evt) == -1) 4999 throw new Exception("epoll_ctl"); 5000 } else { 5001 // initial registration 5002 *registered = true ; 5003 int fd = source.handle; 5004 epoll_event evt; 5005 evt.events = e | EPOLLONESHOT; 5006 evt.data.ptr = cast(void*) f; 5007 if(epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &evt) == -1) 5008 throw new Exception("epoll_ctl"); 5009 } 5010 }; 5011 5012 Fiber.yield(); 5013 5014 f.setPostYield(null); 5015 } else version(Windows) { 5016 Fiber.yield(); 5017 } 5018 else static assert(0); 5019 } 5020 5021 version(cgi_use_fiber) 5022 void unregisterSource(Socket s) { 5023 version(linux) { 5024 epoll_event evt; 5025 epoll_ctl(epfd, EPOLL_CTL_DEL, s.handle(), &evt); 5026 } else version(Windows) { 5027 // intentionally blank 5028 } 5029 else static assert(0); 5030 } 5031 5032 // it is a class primarily for reference semantics 5033 // I might change this interface 5034 /// This is NOT ACTUALLY an input range! It is too different. Historical mistake kinda. 5035 class BufferedInputRange { 5036 version(Posix) 5037 this(int source, ubyte[] buffer = null) { 5038 this(new Socket(cast(socket_t) source, AddressFamily.INET), buffer); 5039 } 5040 5041 this(Socket source, ubyte[] buffer = null, bool allowGrowth = true) { 5042 // if they connect but never send stuff to us, we don't want it wasting the process 5043 // so setting a time out 5044 version(cgi_use_fiber) 5045 source.blocking = false; 5046 else 5047 source.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(3)); 5048 5049 this.source = source; 5050 if(buffer is null) { 5051 underlyingBuffer = new ubyte[4096]; 5052 this.allowGrowth = true; 5053 } else { 5054 underlyingBuffer = buffer; 5055 this.allowGrowth = allowGrowth; 5056 } 5057 5058 assert(underlyingBuffer.length); 5059 5060 // we assume view.ptr is always inside underlyingBuffer 5061 view = underlyingBuffer[0 .. 0]; 5062 5063 popFront(); // prime 5064 } 5065 5066 version(cgi_use_fiber) { 5067 bool registered; 5068 } 5069 5070 void dispose() { 5071 version(cgi_use_fiber) { 5072 if(registered) 5073 unregisterSource(source); 5074 } 5075 } 5076 5077 /** 5078 A slight difference from regular ranges is you can give it the maximum 5079 number of bytes to consume. 5080 5081 IMPORTANT NOTE: the default is to consume nothing, so if you don't call 5082 consume() yourself and use a regular foreach, it will infinitely loop! 5083 5084 The default is to do what a normal range does, and consume the whole buffer 5085 and wait for additional input. 5086 5087 You can also specify 0, to append to the buffer, or any other number 5088 to remove the front n bytes and wait for more. 5089 */ 5090 void popFront(size_t maxBytesToConsume = 0 /*size_t.max*/, size_t minBytesToSettleFor = 0, bool skipConsume = false) { 5091 if(sourceClosed) 5092 throw new ConnectionClosedException("can't get any more data from a closed source"); 5093 if(!skipConsume) 5094 consume(maxBytesToConsume); 5095 5096 // we might have to grow the buffer 5097 if(minBytesToSettleFor > underlyingBuffer.length || view.length == underlyingBuffer.length) { 5098 if(allowGrowth) { 5099 //import std.stdio; writeln("growth"); 5100 auto viewStart = view.ptr - underlyingBuffer.ptr; 5101 size_t growth = 4096; 5102 // make sure we have enough for what we're being asked for 5103 if(minBytesToSettleFor > 0 && minBytesToSettleFor - underlyingBuffer.length > growth) 5104 growth = minBytesToSettleFor - underlyingBuffer.length; 5105 //import std.stdio; writeln(underlyingBuffer.length, " ", viewStart, " ", view.length, " ", growth, " ", minBytesToSettleFor, " ", minBytesToSettleFor - underlyingBuffer.length); 5106 underlyingBuffer.length += growth; 5107 view = underlyingBuffer[viewStart .. view.length]; 5108 } else 5109 throw new Exception("No room left in the buffer"); 5110 } 5111 5112 do { 5113 auto freeSpace = underlyingBuffer[view.ptr - underlyingBuffer.ptr + view.length .. $]; 5114 try_again: 5115 auto ret = source.receive(freeSpace); 5116 if(ret == Socket.ERROR) { 5117 if(wouldHaveBlocked()) { 5118 version(cgi_use_fiber) { 5119 registerEventWakeup(®istered, source, WakeupEvent.Read); 5120 goto try_again; 5121 } else { 5122 // gonna treat a timeout here as a close 5123 sourceClosed = true; 5124 return; 5125 } 5126 } 5127 version(Posix) { 5128 import core.stdc.errno; 5129 if(errno == EINTR || errno == EAGAIN) { 5130 goto try_again; 5131 } 5132 if(errno == ECONNRESET) { 5133 sourceClosed = true; 5134 return; 5135 } 5136 } 5137 throw new Exception(lastSocketError); // FIXME 5138 } 5139 if(ret == 0) { 5140 sourceClosed = true; 5141 return; 5142 } 5143 5144 //import std.stdio; writeln(view.ptr); writeln(underlyingBuffer.ptr); writeln(view.length, " ", ret, " = ", view.length + ret); 5145 view = underlyingBuffer[view.ptr - underlyingBuffer.ptr .. view.length + ret]; 5146 //import std.stdio; writeln(cast(string) view); 5147 } while(view.length < minBytesToSettleFor); 5148 } 5149 5150 /// Removes n bytes from the front of the buffer, and returns the new buffer slice. 5151 /// You might want to idup the data you are consuming if you store it, since it may 5152 /// be overwritten on the new popFront. 5153 /// 5154 /// You do not need to call this if you always want to wait for more data when you 5155 /// consume some. 5156 ubyte[] consume(size_t bytes) { 5157 //import std.stdio; writeln("consuime ", bytes, "/", view.length); 5158 view = view[bytes > $ ? $ : bytes .. $]; 5159 if(view.length == 0) { 5160 view = underlyingBuffer[0 .. 0]; // go ahead and reuse the beginning 5161 /* 5162 writeln("HERE"); 5163 popFront(0, 0, true); // try to load more if we can, checks if the source is closed 5164 writeln(cast(string)front); 5165 writeln("DONE"); 5166 */ 5167 } 5168 return front; 5169 } 5170 5171 bool empty() { 5172 return sourceClosed && view.length == 0; 5173 } 5174 5175 ubyte[] front() { 5176 return view; 5177 } 5178 5179 invariant() { 5180 assert(view.ptr >= underlyingBuffer.ptr); 5181 // it should never be equal, since if that happens view ought to be empty, and thus reusing the buffer 5182 assert(view.ptr < underlyingBuffer.ptr + underlyingBuffer.length); 5183 } 5184 5185 ubyte[] underlyingBuffer; 5186 bool allowGrowth; 5187 ubyte[] view; 5188 Socket source; 5189 bool sourceClosed; 5190 } 5191 5192 private class FakeSocketForStdin : Socket { 5193 import std.stdio; 5194 5195 this() { 5196 5197 } 5198 5199 private bool closed; 5200 5201 override ptrdiff_t receive(scope void[] buffer, std.socket.SocketFlags) @trusted { 5202 if(closed) 5203 throw new Exception("Closed"); 5204 return stdin.rawRead(buffer).length; 5205 } 5206 5207 override ptrdiff_t send(const scope void[] buffer, std.socket.SocketFlags) @trusted { 5208 if(closed) 5209 throw new Exception("Closed"); 5210 stdout.rawWrite(buffer); 5211 return buffer.length; 5212 } 5213 5214 override void close() @trusted scope { 5215 (cast(void delegate() @nogc nothrow) &realClose)(); 5216 } 5217 5218 override void shutdown(SocketShutdown s) { 5219 // FIXME 5220 } 5221 5222 override void setOption(SocketOptionLevel, SocketOption, scope void[]) {} 5223 override void setOption(SocketOptionLevel, SocketOption, Duration) {} 5224 5225 override @property @trusted Address remoteAddress() { return null; } 5226 override @property @trusted Address localAddress() { return null; } 5227 5228 void realClose() { 5229 closed = true; 5230 try { 5231 stdin.close(); 5232 stdout.close(); 5233 } catch(Exception e) { 5234 5235 } 5236 } 5237 } 5238 5239 import core.sync.semaphore; 5240 import core.atomic; 5241 5242 /** 5243 To use this thing: 5244 5245 --- 5246 void handler(Socket s) { do something... } 5247 auto manager = new ListeningConnectionManager("127.0.0.1", 80, &handler, &delegateThatDropsPrivileges); 5248 manager.listen(); 5249 --- 5250 5251 The 4th parameter is optional. 5252 5253 I suggest you use BufferedInputRange(connection) to handle the input. As a packet 5254 comes in, you will get control. You can just continue; though to fetch more. 5255 5256 5257 FIXME: should I offer an event based async thing like netman did too? Yeah, probably. 5258 */ 5259 class ListeningConnectionManager { 5260 Semaphore semaphore; 5261 Socket[256] queue; 5262 shared(ubyte) nextIndexFront; 5263 ubyte nextIndexBack; 5264 shared(int) queueLength; 5265 5266 Socket acceptCancelable() { 5267 version(Posix) { 5268 import core.sys.posix.sys.select; 5269 fd_set read_fds; 5270 FD_ZERO(&read_fds); 5271 FD_SET(listener.handle, &read_fds); 5272 if(cancelfd != -1) 5273 FD_SET(cancelfd, &read_fds); 5274 auto max = listener.handle > cancelfd ? listener.handle : cancelfd; 5275 auto ret = select(max + 1, &read_fds, null, null, null); 5276 if(ret == -1) { 5277 import core.stdc.errno; 5278 if(errno == EINTR) 5279 return null; 5280 else 5281 throw new Exception("wtf select"); 5282 } 5283 5284 if(cancelfd != -1 && FD_ISSET(cancelfd, &read_fds)) { 5285 return null; 5286 } 5287 5288 if(FD_ISSET(listener.handle, &read_fds)) 5289 return listener.accept(); 5290 5291 return null; 5292 } else 5293 return listener.accept(); // FIXME: check the cancel flag! 5294 } 5295 5296 int defaultNumberOfThreads() { 5297 import std.parallelism; 5298 version(cgi_use_fiber) { 5299 return totalCPUs * 1 + 1; 5300 } else { 5301 // I times 4 here because there's a good chance some will be blocked on i/o. 5302 return totalCPUs * 4; 5303 } 5304 5305 } 5306 5307 void listen() { 5308 shared(int) loopBroken; 5309 5310 version(Posix) { 5311 import core.sys.posix.signal; 5312 signal(SIGPIPE, SIG_IGN); 5313 } 5314 5315 version(linux) { 5316 if(cancelfd == -1) 5317 cancelfd = eventfd(0, 0); 5318 } 5319 5320 version(cgi_no_threads) { 5321 // NEVER USE THIS 5322 // it exists only for debugging and other special occasions 5323 5324 // the thread mode is faster and less likely to stall the whole 5325 // thing when a request is slow 5326 while(!loopBroken && !globalStopFlag) { 5327 auto sn = acceptCancelable(); 5328 if(sn is null) continue; 5329 cloexec(sn); 5330 try { 5331 handler(sn); 5332 } catch(Exception e) { 5333 // if a connection goes wrong, we want to just say no, but try to carry on unless it is an Error of some sort (in which case, we'll die. You might want an external helper program to revive the server when it dies) 5334 sn.close(); 5335 } 5336 } 5337 } else { 5338 5339 if(useFork) { 5340 version(linux) { 5341 //asm { int 3; } 5342 fork(); 5343 } 5344 } 5345 5346 version(cgi_use_fiber) { 5347 5348 version(Windows) { 5349 listener.accept(); 5350 } 5351 5352 WorkerThread[] threads = new WorkerThread[](numberOfThreads); 5353 foreach(i, ref thread; threads) { 5354 thread = new WorkerThread(this, handler, cast(int) i); 5355 thread.start(); 5356 } 5357 5358 bool fiber_crash_check() { 5359 bool hasAnyRunning; 5360 foreach(thread; threads) { 5361 if(!thread.isRunning) { 5362 thread.join(); 5363 } else hasAnyRunning = true; 5364 } 5365 5366 return (!hasAnyRunning); 5367 } 5368 5369 5370 while(!globalStopFlag) { 5371 Thread.sleep(1.seconds); 5372 if(fiber_crash_check()) 5373 break; 5374 } 5375 5376 } else { 5377 semaphore = new Semaphore(); 5378 5379 ConnectionThread[] threads = new ConnectionThread[](numberOfThreads); 5380 foreach(i, ref thread; threads) { 5381 thread = new ConnectionThread(this, handler, cast(int) i); 5382 thread.start(); 5383 } 5384 5385 while(!loopBroken && !globalStopFlag) { 5386 Socket sn; 5387 5388 bool crash_check() { 5389 bool hasAnyRunning; 5390 foreach(thread; threads) { 5391 if(!thread.isRunning) { 5392 thread.join(); 5393 } else hasAnyRunning = true; 5394 } 5395 5396 return (!hasAnyRunning); 5397 } 5398 5399 5400 void accept_new_connection() { 5401 sn = acceptCancelable(); 5402 if(sn is null) return; 5403 cloexec(sn); 5404 if(tcp) { 5405 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5406 // on the socket because we do some buffering internally. I think this helps, 5407 // certainly does for small requests, and I think it does for larger ones too 5408 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5409 5410 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5411 } 5412 } 5413 5414 void existing_connection_new_data() { 5415 // wait until a slot opens up 5416 //int waited = 0; 5417 while(queueLength >= queue.length) { 5418 Thread.sleep(1.msecs); 5419 //waited ++; 5420 } 5421 //if(waited) {import std.stdio; writeln(waited);} 5422 synchronized(this) { 5423 queue[nextIndexBack] = sn; 5424 nextIndexBack++; 5425 atomicOp!"+="(queueLength, 1); 5426 } 5427 semaphore.notify(); 5428 } 5429 5430 5431 accept_new_connection(); 5432 if(sn !is null) 5433 existing_connection_new_data(); 5434 else if(sn is null && globalStopFlag) { 5435 foreach(thread; threads) { 5436 semaphore.notify(); 5437 } 5438 Thread.sleep(50.msecs); 5439 } 5440 5441 if(crash_check()) 5442 break; 5443 } 5444 } 5445 5446 // FIXME: i typically stop this with ctrl+c which never 5447 // actually gets here. i need to do a sigint handler. 5448 if(cleanup) 5449 cleanup(); 5450 } 5451 } 5452 5453 //version(linux) 5454 //int epoll_fd; 5455 5456 bool tcp; 5457 void delegate() cleanup; 5458 5459 private void function(Socket) fhandler; 5460 private void dg_handler(Socket s) { 5461 fhandler(s); 5462 } 5463 this(string host, ushort port, void function(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5464 fhandler = handler; 5465 this(host, port, &dg_handler, dropPrivs, useFork, numberOfThreads); 5466 } 5467 5468 this(string host, ushort port, void delegate(Socket) handler, void delegate() dropPrivs = null, bool useFork = cgi_use_fork_default, int numberOfThreads = 0) { 5469 this.handler = handler; 5470 this.useFork = useFork; 5471 this.numberOfThreads = numberOfThreads ? numberOfThreads : defaultNumberOfThreads(); 5472 5473 listener = startListening(host, port, tcp, cleanup, 128, dropPrivs); 5474 5475 version(cgi_use_fiber) 5476 if(useFork) 5477 listener.blocking = false; 5478 5479 // this is the UI control thread and thus gets more priority 5480 Thread.getThis.priority = Thread.PRIORITY_MAX; 5481 } 5482 5483 Socket listener; 5484 void delegate(Socket) handler; 5485 5486 immutable bool useFork; 5487 int numberOfThreads; 5488 } 5489 5490 Socket startListening(string host, ushort port, ref bool tcp, ref void delegate() cleanup, int backQueue, void delegate() dropPrivs) { 5491 Socket listener; 5492 if(host.startsWith("unix:")) { 5493 version(Posix) { 5494 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5495 cloexec(listener); 5496 string filename = host["unix:".length .. $].idup; 5497 listener.bind(new UnixAddress(filename)); 5498 cleanup = delegate() { 5499 listener.close(); 5500 import std.file; 5501 remove(filename); 5502 }; 5503 tcp = false; 5504 } else { 5505 throw new Exception("unix sockets not supported on this system"); 5506 } 5507 } else if(host.startsWith("abstract:")) { 5508 version(linux) { 5509 listener = new Socket(AddressFamily.UNIX, SocketType.STREAM); 5510 cloexec(listener); 5511 string filename = "\0" ~ host["abstract:".length .. $]; 5512 import std.stdio; stderr.writeln("Listening to abstract unix domain socket: ", host["abstract:".length .. $]); 5513 listener.bind(new UnixAddress(filename)); 5514 tcp = false; 5515 } else { 5516 throw new Exception("abstract unix sockets not supported on this system"); 5517 } 5518 } else { 5519 version(cgi_use_fiber) { 5520 version(Windows) 5521 listener = new PseudoblockingOverlappedSocket(AddressFamily.INET, SocketType.STREAM); 5522 else 5523 listener = new TcpSocket(); 5524 } else { 5525 listener = new TcpSocket(); 5526 } 5527 cloexec(listener); 5528 listener.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); 5529 listener.bind(host.length ? parseAddress(host, port) : new InternetAddress(port)); 5530 cleanup = delegate() { 5531 listener.close(); 5532 }; 5533 tcp = true; 5534 } 5535 5536 listener.listen(backQueue); 5537 5538 if (dropPrivs !is null) // can be null, backwards compatibility 5539 dropPrivs(); 5540 5541 return listener; 5542 } 5543 5544 // helper function to send a lot to a socket. Since this blocks for the buffer (possibly several times), you should probably call it in a separate thread or something. 5545 void sendAll(Socket s, const(void)[] data, string file = __FILE__, size_t line = __LINE__) { 5546 if(data.length == 0) return; 5547 ptrdiff_t amount; 5548 //import std.stdio; writeln("***",cast(string) data,"///"); 5549 do { 5550 amount = s.send(data); 5551 if(amount == Socket.ERROR) { 5552 version(cgi_use_fiber) { 5553 if(wouldHaveBlocked()) { 5554 bool registered = true; 5555 registerEventWakeup(®istered, s, WakeupEvent.Write); 5556 continue; 5557 } 5558 } 5559 throw new ConnectionException(s, lastSocketError, file, line); 5560 } 5561 assert(amount > 0); 5562 5563 data = data[amount .. $]; 5564 } while(data.length); 5565 } 5566 5567 class ConnectionException : Exception { 5568 Socket socket; 5569 this(Socket s, string msg, string file = __FILE__, size_t line = __LINE__) { 5570 this.socket = s; 5571 super(msg, file, line); 5572 } 5573 } 5574 5575 alias void delegate(Socket) CMT; 5576 5577 import core.thread; 5578 /+ 5579 cgi.d now uses a hybrid of event i/o and threads at the top level. 5580 5581 Top level thread is responsible for accepting sockets and selecting on them. 5582 5583 It then indicates to a child that a request is pending, and any random worker 5584 thread that is free handles it. It goes into blocking mode and handles that 5585 http request to completion. 5586 5587 At that point, it goes back into the waiting queue. 5588 5589 5590 This concept is only implemented on Linux. On all other systems, it still 5591 uses the worker threads and semaphores (which is perfectly fine for a lot of 5592 things! Just having a great number of keep-alive connections will break that.) 5593 5594 5595 So the algorithm is: 5596 5597 select(accept, event, pending) 5598 if accept -> send socket to free thread, if any. if not, add socket to queue 5599 if event -> send the signaling thread a socket from the queue, if not, mark it free 5600 - event might block until it can be *written* to. it is a fifo sending socket fds! 5601 5602 A worker only does one http request at a time, then signals its availability back to the boss. 5603 5604 The socket the worker was just doing should be added to the one-off epoll read. If it is closed, 5605 great, we can get rid of it. Otherwise, it is considered `pending`. The *kernel* manages that; the 5606 actual FD will not be kept out here. 5607 5608 So: 5609 queue = sockets we know are ready to read now, but no worker thread is available 5610 idle list = worker threads not doing anything else. they signal back and forth 5611 5612 the workers all read off the event fd. This is the semaphore wait 5613 5614 the boss waits on accept or other sockets read events (one off! and level triggered). If anything happens wrt ready read, 5615 it puts it in the queue and writes to the event fd. 5616 5617 The child could put the socket back in the epoll thing itself. 5618 5619 The child needs to be able to gracefully handle being given a socket that just closed with no work. 5620 +/ 5621 class ConnectionThread : Thread { 5622 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 5623 this.lcm = lcm; 5624 this.dg = dg; 5625 this.myThreadNumber = myThreadNumber; 5626 super(&run); 5627 } 5628 5629 void run() { 5630 while(true) { 5631 // so if there's a bunch of idle keep-alive connections, it can 5632 // consume all the worker threads... just sitting there. 5633 lcm.semaphore.wait(); 5634 if(globalStopFlag) 5635 return; 5636 Socket socket; 5637 synchronized(lcm) { 5638 auto idx = lcm.nextIndexFront; 5639 socket = lcm.queue[idx]; 5640 lcm.queue[idx] = null; 5641 atomicOp!"+="(lcm.nextIndexFront, 1); 5642 atomicOp!"-="(lcm.queueLength, 1); 5643 } 5644 try { 5645 //import std.stdio; writeln(myThreadNumber, " taking it"); 5646 dg(socket); 5647 /+ 5648 if(socket.isAlive) { 5649 // process it more later 5650 version(linux) { 5651 import core.sys.linux.epoll; 5652 epoll_event ev; 5653 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 5654 ev.data.fd = socket.handle; 5655 import std.stdio; writeln("adding"); 5656 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_ADD, socket.handle, &ev) == -1) { 5657 if(errno == EEXIST) { 5658 ev.events = EPOLLIN | EPOLLONESHOT | EPOLLET; 5659 ev.data.fd = socket.handle; 5660 if(epoll_ctl(lcm.epoll_fd, EPOLL_CTL_MOD, socket.handle, &ev) == -1) 5661 throw new Exception("epoll_ctl " ~ to!string(errno)); 5662 } else 5663 throw new Exception("epoll_ctl " ~ to!string(errno)); 5664 } 5665 //import std.stdio; writeln("keep alive"); 5666 // writing to this private member is to prevent the GC from closing my precious socket when I'm trying to use it later 5667 __traits(getMember, socket, "sock") = cast(socket_t) -1; 5668 } else { 5669 continue; // hope it times out in a reasonable amount of time... 5670 } 5671 } 5672 +/ 5673 } catch(ConnectionClosedException e) { 5674 // can just ignore this, it is fairly normal 5675 socket.close(); 5676 } catch(Throwable e) { 5677 import std.stdio; stderr.rawWrite(e.toString); stderr.rawWrite("\n"); 5678 socket.close(); 5679 } 5680 } 5681 } 5682 5683 ListeningConnectionManager lcm; 5684 CMT dg; 5685 int myThreadNumber; 5686 } 5687 5688 version(cgi_use_fiber) 5689 class WorkerThread : Thread { 5690 this(ListeningConnectionManager lcm, CMT dg, int myThreadNumber) { 5691 this.lcm = lcm; 5692 this.dg = dg; 5693 this.myThreadNumber = myThreadNumber; 5694 super(&run); 5695 } 5696 5697 version(Windows) 5698 void run() { 5699 auto timeout = INFINITE; 5700 PseudoblockingOverlappedSocket key; 5701 OVERLAPPED* overlapped; 5702 DWORD bytes; 5703 while(!globalStopFlag && GetQueuedCompletionStatus(iocp, &bytes, cast(PULONG_PTR) &key, &overlapped, timeout)) { 5704 if(key is null) 5705 continue; 5706 key.lastAnswer = bytes; 5707 if(key.fiber) { 5708 key.fiber.proceed(); 5709 } else { 5710 // we have a new connection, issue the first receive on it and issue the next accept 5711 5712 auto sn = key.accepted; 5713 5714 key.accept(); 5715 5716 cloexec(sn); 5717 if(lcm.tcp) { 5718 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5719 // on the socket because we do some buffering internally. I think this helps, 5720 // certainly does for small requests, and I think it does for larger ones too 5721 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5722 5723 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5724 } 5725 5726 dg(sn); 5727 } 5728 } 5729 //SleepEx(INFINITE, TRUE); 5730 } 5731 5732 version(linux) 5733 void run() { 5734 5735 import core.sys.linux.epoll; 5736 epfd = epoll_create1(EPOLL_CLOEXEC); 5737 if(epfd == -1) 5738 throw new Exception("epoll_create1 " ~ to!string(errno)); 5739 scope(exit) { 5740 import core.sys.posix.unistd; 5741 close(epfd); 5742 } 5743 5744 { 5745 epoll_event ev; 5746 ev.events = EPOLLIN; 5747 ev.data.fd = cancelfd; 5748 epoll_ctl(epfd, EPOLL_CTL_ADD, cancelfd, &ev); 5749 } 5750 5751 epoll_event ev; 5752 ev.events = EPOLLIN | EPOLLEXCLUSIVE; // EPOLLEXCLUSIVE is only available on kernels since like 2017 but that's prolly good enough. 5753 ev.data.fd = lcm.listener.handle; 5754 if(epoll_ctl(epfd, EPOLL_CTL_ADD, lcm.listener.handle, &ev) == -1) 5755 throw new Exception("epoll_ctl " ~ to!string(errno)); 5756 5757 5758 5759 while(!globalStopFlag) { 5760 Socket sn; 5761 5762 epoll_event[64] events; 5763 auto nfds = epoll_wait(epfd, events.ptr, events.length, -1); 5764 if(nfds == -1) { 5765 if(errno == EINTR) 5766 continue; 5767 throw new Exception("epoll_wait " ~ to!string(errno)); 5768 } 5769 5770 foreach(idx; 0 .. nfds) { 5771 auto flags = events[idx].events; 5772 5773 if(cast(size_t) events[idx].data.ptr == cast(size_t) cancelfd) { 5774 globalStopFlag = true; 5775 //import std.stdio; writeln("exit heard"); 5776 break; 5777 } else if(cast(size_t) events[idx].data.ptr == cast(size_t) lcm.listener.handle) { 5778 //import std.stdio; writeln(myThreadNumber, " woken up ", flags); 5779 // this try/catch is because it is set to non-blocking mode 5780 // and Phobos' stupid api throws an exception instead of returning 5781 // if it would block. Why would it block? because a forked process 5782 // might have beat us to it, but the wakeup event thundered our herds. 5783 try 5784 sn = lcm.listener.accept(); // don't need to do the acceptCancelable here since the epoll checks it better 5785 catch(SocketAcceptException e) { continue; } 5786 5787 cloexec(sn); 5788 if(lcm.tcp) { 5789 // disable Nagle's algorithm to avoid a 40ms delay when we send/recv 5790 // on the socket because we do some buffering internally. I think this helps, 5791 // certainly does for small requests, and I think it does for larger ones too 5792 sn.setOption(SocketOptionLevel.TCP, SocketOption.TCP_NODELAY, 1); 5793 5794 sn.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"seconds"(10)); 5795 } 5796 5797 dg(sn); 5798 } else { 5799 if(cast(size_t) events[idx].data.ptr < 1024) { 5800 throw new Exception("this doesn't look like a fiber pointer..."); 5801 } 5802 auto fiber = cast(CgiFiber) events[idx].data.ptr; 5803 fiber.proceed(); 5804 } 5805 } 5806 } 5807 } 5808 5809 ListeningConnectionManager lcm; 5810 CMT dg; 5811 int myThreadNumber; 5812 } 5813 5814 5815 /* Done with network helper */ 5816 5817 /* Helpers for doing temporary files. Used both here and in web.d */ 5818 5819 version(Windows) { 5820 import core.sys.windows.windows; 5821 extern(Windows) DWORD GetTempPathW(DWORD, LPWSTR); 5822 alias GetTempPathW GetTempPath; 5823 } 5824 5825 version(Posix) { 5826 static import linux = core.sys.posix.unistd; 5827 } 5828 5829 string getTempDirectory() { 5830 string path; 5831 version(Windows) { 5832 wchar[1024] buffer; 5833 auto len = GetTempPath(1024, buffer.ptr); 5834 if(len == 0) 5835 throw new Exception("couldn't find a temporary path"); 5836 5837 auto b = buffer[0 .. len]; 5838 5839 path = to!string(b); 5840 } else 5841 path = "/tmp/"; 5842 5843 return path; 5844 } 5845 5846 5847 // I like std.date. These functions help keep my old code and data working with phobos changing. 5848 5849 long sysTimeToDTime(in SysTime sysTime) { 5850 return convert!("hnsecs", "msecs")(sysTime.stdTime - 621355968000000000L); 5851 } 5852 5853 long dateTimeToDTime(in DateTime dt) { 5854 return sysTimeToDTime(cast(SysTime) dt); 5855 } 5856 5857 long getUtcTime() { // renamed primarily to avoid conflict with std.date itself 5858 return sysTimeToDTime(Clock.currTime(UTC())); 5859 } 5860 5861 // NOTE: new SimpleTimeZone(minutes); can perhaps work with the getTimezoneOffset() JS trick 5862 SysTime dTimeToSysTime(long dTime, immutable TimeZone tz = null) { 5863 immutable hnsecs = convert!("msecs", "hnsecs")(dTime) + 621355968000000000L; 5864 return SysTime(hnsecs, tz); 5865 } 5866 5867 5868 5869 // this is a helper to read HTTP transfer-encoding: chunked responses 5870 immutable(ubyte[]) dechunk(BufferedInputRange ir) { 5871 immutable(ubyte)[] ret; 5872 5873 another_chunk: 5874 // If here, we are at the beginning of a chunk. 5875 auto a = ir.front(); 5876 int chunkSize; 5877 int loc = locationOf(a, "\r\n"); 5878 while(loc == -1) { 5879 ir.popFront(); 5880 a = ir.front(); 5881 loc = locationOf(a, "\r\n"); 5882 } 5883 5884 string hex; 5885 hex = ""; 5886 for(int i = 0; i < loc; i++) { 5887 char c = a[i]; 5888 if(c >= 'A' && c <= 'Z') 5889 c += 0x20; 5890 if((c >= '0' && c <= '9') || (c >= 'a' && c <= 'z')) { 5891 hex ~= c; 5892 } else { 5893 break; 5894 } 5895 } 5896 5897 assert(hex.length); 5898 5899 int power = 1; 5900 int size = 0; 5901 foreach(cc1; retro(hex)) { 5902 dchar cc = cc1; 5903 if(cc >= 'a' && cc <= 'z') 5904 cc -= 0x20; 5905 int val = 0; 5906 if(cc >= '0' && cc <= '9') 5907 val = cc - '0'; 5908 else 5909 val = cc - 'A' + 10; 5910 5911 size += power * val; 5912 power *= 16; 5913 } 5914 5915 chunkSize = size; 5916 assert(size >= 0); 5917 5918 if(loc + 2 > a.length) { 5919 ir.popFront(0, a.length + loc + 2); 5920 a = ir.front(); 5921 } 5922 5923 a = ir.consume(loc + 2); 5924 5925 if(chunkSize == 0) { // we're done with the response 5926 // if we got here, will change must be true.... 5927 more_footers: 5928 loc = locationOf(a, "\r\n"); 5929 if(loc == -1) { 5930 ir.popFront(); 5931 a = ir.front; 5932 goto more_footers; 5933 } else { 5934 assert(loc == 0); 5935 ir.consume(loc + 2); 5936 goto finish; 5937 } 5938 } else { 5939 // if we got here, will change must be true.... 5940 if(a.length < chunkSize + 2) { 5941 ir.popFront(0, chunkSize + 2); 5942 a = ir.front(); 5943 } 5944 5945 ret ~= (a[0..chunkSize]); 5946 5947 if(!(a.length > chunkSize + 2)) { 5948 ir.popFront(0, chunkSize + 2); 5949 a = ir.front(); 5950 } 5951 assert(a[chunkSize] == 13); 5952 assert(a[chunkSize+1] == 10); 5953 a = ir.consume(chunkSize + 2); 5954 chunkSize = 0; 5955 goto another_chunk; 5956 } 5957 5958 finish: 5959 return ret; 5960 } 5961 5962 // I want to be able to get data from multiple sources the same way... 5963 interface ByChunkRange { 5964 bool empty(); 5965 void popFront(); 5966 const(ubyte)[] front(); 5967 } 5968 5969 ByChunkRange byChunk(const(ubyte)[] data) { 5970 return new class ByChunkRange { 5971 override bool empty() { 5972 return !data.length; 5973 } 5974 5975 override void popFront() { 5976 if(data.length > 4096) 5977 data = data[4096 .. $]; 5978 else 5979 data = null; 5980 } 5981 5982 override const(ubyte)[] front() { 5983 return data[0 .. $ > 4096 ? 4096 : $]; 5984 } 5985 }; 5986 } 5987 5988 ByChunkRange byChunk(BufferedInputRange ir, size_t atMost) { 5989 const(ubyte)[] f; 5990 5991 f = ir.front; 5992 if(f.length > atMost) 5993 f = f[0 .. atMost]; 5994 5995 return new class ByChunkRange { 5996 override bool empty() { 5997 return atMost == 0; 5998 } 5999 6000 override const(ubyte)[] front() { 6001 return f; 6002 } 6003 6004 override void popFront() { 6005 ir.consume(f.length); 6006 atMost -= f.length; 6007 auto a = ir.front(); 6008 6009 if(a.length <= atMost) { 6010 f = a; 6011 atMost -= a.length; 6012 a = ir.consume(a.length); 6013 if(atMost != 0) 6014 ir.popFront(); 6015 if(f.length == 0) { 6016 f = ir.front(); 6017 } 6018 } else { 6019 // we actually have *more* here than we need.... 6020 f = a[0..atMost]; 6021 atMost = 0; 6022 ir.consume(atMost); 6023 } 6024 } 6025 }; 6026 } 6027 6028 version(cgi_with_websocket) { 6029 // http://tools.ietf.org/html/rfc6455 6030 6031 /** 6032 WEBSOCKET SUPPORT: 6033 6034 Full example: 6035 --- 6036 import arsd.cgi; 6037 6038 void websocketEcho(Cgi cgi) { 6039 if(cgi.websocketRequested()) { 6040 if(cgi.origin != "http://arsdnet.net") 6041 throw new Exception("bad origin"); 6042 auto websocket = cgi.acceptWebsocket(); 6043 6044 websocket.send("hello"); 6045 websocket.send(" world!"); 6046 6047 auto msg = websocket.recv(); 6048 while(msg.opcode != WebSocketOpcode.close) { 6049 if(msg.opcode == WebSocketOpcode.text) { 6050 websocket.send(msg.textData); 6051 } else if(msg.opcode == WebSocketOpcode.binary) { 6052 websocket.send(msg.data); 6053 } 6054 6055 msg = websocket.recv(); 6056 } 6057 6058 websocket.close(); 6059 } else assert(0, "i want a web socket!"); 6060 } 6061 6062 mixin GenericMain!websocketEcho; 6063 --- 6064 */ 6065 6066 class WebSocket { 6067 Cgi cgi; 6068 6069 private this(Cgi cgi) { 6070 this.cgi = cgi; 6071 6072 Socket socket = cgi.idlol.source; 6073 socket.setOption(SocketOptionLevel.SOCKET, SocketOption.RCVTIMEO, dur!"minutes"(5)); 6074 } 6075 6076 // returns true if data available, false if it timed out 6077 bool recvAvailable(Duration timeout = dur!"msecs"(0)) { 6078 if(!waitForNextMessageWouldBlock()) 6079 return true; 6080 if(isDataPending(timeout)) 6081 return true; // this is kinda a lie. 6082 6083 return false; 6084 } 6085 6086 public bool lowLevelReceive() { 6087 auto bfr = cgi.idlol; 6088 top: 6089 auto got = bfr.front; 6090 if(got.length) { 6091 if(receiveBuffer.length < receiveBufferUsedLength + got.length) 6092 receiveBuffer.length += receiveBufferUsedLength + got.length; 6093 6094 receiveBuffer[receiveBufferUsedLength .. receiveBufferUsedLength + got.length] = got[]; 6095 receiveBufferUsedLength += got.length; 6096 bfr.consume(got.length); 6097 6098 return true; 6099 } 6100 6101 if(bfr.sourceClosed) 6102 return false; 6103 6104 bfr.popFront(0); 6105 if(bfr.sourceClosed) 6106 return false; 6107 goto top; 6108 } 6109 6110 6111 bool isDataPending(Duration timeout = 0.seconds) { 6112 Socket socket = cgi.idlol.source; 6113 6114 auto check = new SocketSet(); 6115 check.add(socket); 6116 6117 auto got = Socket.select(check, null, null, timeout); 6118 if(got > 0) 6119 return true; 6120 return false; 6121 } 6122 6123 // note: this blocks 6124 WebSocketFrame recv() { 6125 return waitForNextMessage(); 6126 } 6127 6128 6129 6130 6131 private void llclose() { 6132 cgi.close(); 6133 } 6134 6135 private void llsend(ubyte[] data) { 6136 cgi.write(data); 6137 cgi.flush(); 6138 } 6139 6140 void unregisterActiveSocket(WebSocket) {} 6141 6142 /* copy/paste section { */ 6143 6144 private int readyState_; 6145 private ubyte[] receiveBuffer; 6146 private size_t receiveBufferUsedLength; 6147 6148 private Config config; 6149 6150 enum CONNECTING = 0; /// Socket has been created. The connection is not yet open. 6151 enum OPEN = 1; /// The connection is open and ready to communicate. 6152 enum CLOSING = 2; /// The connection is in the process of closing. 6153 enum CLOSED = 3; /// The connection is closed or couldn't be opened. 6154 6155 /++ 6156 6157 +/ 6158 /// Group: foundational 6159 static struct Config { 6160 /++ 6161 These control the size of the receive buffer. 6162 6163 It starts at the initial size, will temporarily 6164 balloon up to the maximum size, and will reuse 6165 a buffer up to the likely size. 6166 6167 Anything larger than the maximum size will cause 6168 the connection to be aborted and an exception thrown. 6169 This is to protect you against a peer trying to 6170 exhaust your memory, while keeping the user-level 6171 processing simple. 6172 +/ 6173 size_t initialReceiveBufferSize = 4096; 6174 size_t likelyReceiveBufferSize = 4096; /// ditto 6175 size_t maximumReceiveBufferSize = 10 * 1024 * 1024; /// ditto 6176 6177 /++ 6178 Maximum combined size of a message. 6179 +/ 6180 size_t maximumMessageSize = 10 * 1024 * 1024; 6181 6182 string[string] cookies; /// Cookies to send with the initial request. cookies[name] = value; 6183 string origin; /// Origin URL to send with the handshake, if desired. 6184 string protocol; /// the protocol header, if desired. 6185 6186 int pingFrequency = 5000; /// Amount of time (in msecs) of idleness after which to send an automatic ping 6187 } 6188 6189 /++ 6190 Returns one of [CONNECTING], [OPEN], [CLOSING], or [CLOSED]. 6191 +/ 6192 int readyState() { 6193 return readyState_; 6194 } 6195 6196 /++ 6197 Closes the connection, sending a graceful teardown message to the other side. 6198 +/ 6199 /// Group: foundational 6200 void close(int code = 0, string reason = null) 6201 //in (reason.length < 123) 6202 in { assert(reason.length < 123); } do 6203 { 6204 if(readyState_ != OPEN) 6205 return; // it cool, we done 6206 WebSocketFrame wss; 6207 wss.fin = true; 6208 wss.opcode = WebSocketOpcode.close; 6209 wss.data = cast(ubyte[]) reason.dup; 6210 wss.send(&llsend); 6211 6212 readyState_ = CLOSING; 6213 6214 llclose(); 6215 } 6216 6217 /++ 6218 Sends a ping message to the server. This is done automatically by the library if you set a non-zero [Config.pingFrequency], but you can also send extra pings explicitly as well with this function. 6219 +/ 6220 /// Group: foundational 6221 void ping() { 6222 WebSocketFrame wss; 6223 wss.fin = true; 6224 wss.opcode = WebSocketOpcode.ping; 6225 wss.send(&llsend); 6226 } 6227 6228 // automatically handled.... 6229 void pong() { 6230 WebSocketFrame wss; 6231 wss.fin = true; 6232 wss.opcode = WebSocketOpcode.pong; 6233 wss.send(&llsend); 6234 } 6235 6236 /++ 6237 Sends a text message through the websocket. 6238 +/ 6239 /// Group: foundational 6240 void send(in char[] textData) { 6241 WebSocketFrame wss; 6242 wss.fin = true; 6243 wss.opcode = WebSocketOpcode.text; 6244 wss.data = cast(ubyte[]) textData.dup; 6245 wss.send(&llsend); 6246 } 6247 6248 /++ 6249 Sends a binary message through the websocket. 6250 +/ 6251 /// Group: foundational 6252 void send(in ubyte[] binaryData) { 6253 WebSocketFrame wss; 6254 wss.fin = true; 6255 wss.opcode = WebSocketOpcode.binary; 6256 wss.data = cast(ubyte[]) binaryData.dup; 6257 wss.send(&llsend); 6258 } 6259 6260 /++ 6261 Waits for and returns the next complete message on the socket. 6262 6263 Note that the onmessage function is still called, right before 6264 this returns. 6265 +/ 6266 /// Group: blocking_api 6267 public WebSocketFrame waitForNextMessage() { 6268 do { 6269 auto m = processOnce(); 6270 if(m.populated) 6271 return m; 6272 } while(lowLevelReceive()); 6273 6274 throw new ConnectionClosedException("Websocket receive timed out"); 6275 //return WebSocketFrame.init; // FIXME? maybe. 6276 } 6277 6278 /++ 6279 Tells if [waitForNextMessage] would block. 6280 +/ 6281 /// Group: blocking_api 6282 public bool waitForNextMessageWouldBlock() { 6283 checkAgain: 6284 if(isMessageBuffered()) 6285 return false; 6286 if(!isDataPending()) 6287 return true; 6288 while(isDataPending()) 6289 lowLevelReceive(); 6290 goto checkAgain; 6291 } 6292 6293 /++ 6294 Is there a message in the buffer already? 6295 If `true`, [waitForNextMessage] is guaranteed to return immediately. 6296 If `false`, check [isDataPending] as the next step. 6297 +/ 6298 /// Group: blocking_api 6299 public bool isMessageBuffered() { 6300 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6301 auto s = d; 6302 if(d.length) { 6303 auto orig = d; 6304 auto m = WebSocketFrame.read(d); 6305 // that's how it indicates that it needs more data 6306 if(d !is orig) 6307 return true; 6308 } 6309 6310 return false; 6311 } 6312 6313 private ubyte continuingType; 6314 private ubyte[] continuingData; 6315 //private size_t continuingDataLength; 6316 6317 private WebSocketFrame processOnce() { 6318 ubyte[] d = receiveBuffer[0 .. receiveBufferUsedLength]; 6319 auto s = d; 6320 // FIXME: handle continuation frames more efficiently. it should really just reuse the receive buffer. 6321 WebSocketFrame m; 6322 if(d.length) { 6323 auto orig = d; 6324 m = WebSocketFrame.read(d); 6325 // that's how it indicates that it needs more data 6326 if(d is orig) 6327 return WebSocketFrame.init; 6328 m.unmaskInPlace(); 6329 switch(m.opcode) { 6330 case WebSocketOpcode.continuation: 6331 if(continuingData.length + m.data.length > config.maximumMessageSize) 6332 throw new Exception("message size exceeded"); 6333 6334 continuingData ~= m.data; 6335 if(m.fin) { 6336 if(ontextmessage) 6337 ontextmessage(cast(char[]) continuingData); 6338 if(onbinarymessage) 6339 onbinarymessage(continuingData); 6340 6341 continuingData = null; 6342 } 6343 break; 6344 case WebSocketOpcode.text: 6345 if(m.fin) { 6346 if(ontextmessage) 6347 ontextmessage(m.textData); 6348 } else { 6349 continuingType = m.opcode; 6350 //continuingDataLength = 0; 6351 continuingData = null; 6352 continuingData ~= m.data; 6353 } 6354 break; 6355 case WebSocketOpcode.binary: 6356 if(m.fin) { 6357 if(onbinarymessage) 6358 onbinarymessage(m.data); 6359 } else { 6360 continuingType = m.opcode; 6361 //continuingDataLength = 0; 6362 continuingData = null; 6363 continuingData ~= m.data; 6364 } 6365 break; 6366 case WebSocketOpcode.close: 6367 readyState_ = CLOSED; 6368 if(onclose) 6369 onclose(); 6370 6371 unregisterActiveSocket(this); 6372 break; 6373 case WebSocketOpcode.ping: 6374 pong(); 6375 break; 6376 case WebSocketOpcode.pong: 6377 // just really references it is still alive, nbd. 6378 break; 6379 default: // ignore though i could and perhaps should throw too 6380 } 6381 } 6382 6383 // the recv thing can be invalidated so gotta copy it over ugh 6384 if(d.length) { 6385 m.data = m.data.dup(); 6386 } 6387 6388 import core.stdc.string; 6389 memmove(receiveBuffer.ptr, d.ptr, d.length); 6390 receiveBufferUsedLength = d.length; 6391 6392 return m; 6393 } 6394 6395 private void autoprocess() { 6396 // FIXME 6397 do { 6398 processOnce(); 6399 } while(lowLevelReceive()); 6400 } 6401 6402 6403 void delegate() onclose; /// 6404 void delegate() onerror; /// 6405 void delegate(in char[]) ontextmessage; /// 6406 void delegate(in ubyte[]) onbinarymessage; /// 6407 void delegate() onopen; /// 6408 6409 /++ 6410 6411 +/ 6412 /// Group: browser_api 6413 void onmessage(void delegate(in char[]) dg) { 6414 ontextmessage = dg; 6415 } 6416 6417 /// ditto 6418 void onmessage(void delegate(in ubyte[]) dg) { 6419 onbinarymessage = dg; 6420 } 6421 6422 /* } end copy/paste */ 6423 6424 6425 } 6426 6427 bool websocketRequested(Cgi cgi) { 6428 return 6429 "sec-websocket-key" in cgi.requestHeaders 6430 && 6431 "connection" in cgi.requestHeaders && 6432 cgi.requestHeaders["connection"].asLowerCase().canFind("upgrade") 6433 && 6434 "upgrade" in cgi.requestHeaders && 6435 cgi.requestHeaders["upgrade"].asLowerCase().equal("websocket") 6436 ; 6437 } 6438 6439 WebSocket acceptWebsocket(Cgi cgi) { 6440 assert(!cgi.closed); 6441 assert(!cgi.outputtedResponseData); 6442 cgi.setResponseStatus("101 Switching Protocols"); 6443 cgi.header("Upgrade: WebSocket"); 6444 cgi.header("Connection: upgrade"); 6445 6446 string key = cgi.requestHeaders["sec-websocket-key"]; 6447 key ~= "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; // the defined guid from the websocket spec 6448 6449 import std.digest.sha; 6450 auto hash = sha1Of(key); 6451 auto accept = Base64.encode(hash); 6452 6453 cgi.header(("Sec-WebSocket-Accept: " ~ accept).idup); 6454 6455 cgi.websocketMode = true; 6456 cgi.write(""); 6457 6458 cgi.flush(); 6459 6460 return new WebSocket(cgi); 6461 } 6462 6463 // FIXME get websocket to work on other modes, not just embedded_httpd 6464 6465 /* copy/paste in http2.d { */ 6466 enum WebSocketOpcode : ubyte { 6467 continuation = 0, 6468 text = 1, 6469 binary = 2, 6470 // 3, 4, 5, 6, 7 RESERVED 6471 close = 8, 6472 ping = 9, 6473 pong = 10, 6474 // 11,12,13,14,15 RESERVED 6475 } 6476 6477 public struct WebSocketFrame { 6478 private bool populated; 6479 bool fin; 6480 bool rsv1; 6481 bool rsv2; 6482 bool rsv3; 6483 WebSocketOpcode opcode; // 4 bits 6484 bool masked; 6485 ubyte lengthIndicator; // don't set this when building one to send 6486 ulong realLength; // don't use when sending 6487 ubyte[4] maskingKey; // don't set this when sending 6488 ubyte[] data; 6489 6490 static WebSocketFrame simpleMessage(WebSocketOpcode opcode, void[] data) { 6491 WebSocketFrame msg; 6492 msg.fin = true; 6493 msg.opcode = opcode; 6494 msg.data = cast(ubyte[]) data.dup; 6495 6496 return msg; 6497 } 6498 6499 private void send(scope void delegate(ubyte[]) llsend) { 6500 ubyte[64] headerScratch; 6501 int headerScratchPos = 0; 6502 6503 realLength = data.length; 6504 6505 { 6506 ubyte b1; 6507 b1 |= cast(ubyte) opcode; 6508 b1 |= rsv3 ? (1 << 4) : 0; 6509 b1 |= rsv2 ? (1 << 5) : 0; 6510 b1 |= rsv1 ? (1 << 6) : 0; 6511 b1 |= fin ? (1 << 7) : 0; 6512 6513 headerScratch[0] = b1; 6514 headerScratchPos++; 6515 } 6516 6517 { 6518 headerScratchPos++; // we'll set header[1] at the end of this 6519 auto rlc = realLength; 6520 ubyte b2; 6521 b2 |= masked ? (1 << 7) : 0; 6522 6523 assert(headerScratchPos == 2); 6524 6525 if(realLength > 65535) { 6526 // use 64 bit length 6527 b2 |= 0x7f; 6528 6529 // FIXME: double check endinaness 6530 foreach(i; 0 .. 8) { 6531 headerScratch[2 + 7 - i] = rlc & 0x0ff; 6532 rlc >>>= 8; 6533 } 6534 6535 headerScratchPos += 8; 6536 } else if(realLength > 125) { 6537 // use 16 bit length 6538 b2 |= 0x7e; 6539 6540 // FIXME: double check endinaness 6541 foreach(i; 0 .. 2) { 6542 headerScratch[2 + 1 - i] = rlc & 0x0ff; 6543 rlc >>>= 8; 6544 } 6545 6546 headerScratchPos += 2; 6547 } else { 6548 // use 7 bit length 6549 b2 |= realLength & 0b_0111_1111; 6550 } 6551 6552 headerScratch[1] = b2; 6553 } 6554 6555 //assert(!masked, "masking key not properly implemented"); 6556 if(masked) { 6557 // FIXME: randomize this 6558 headerScratch[headerScratchPos .. headerScratchPos + 4] = maskingKey[]; 6559 headerScratchPos += 4; 6560 6561 // we'll just mask it in place... 6562 int keyIdx = 0; 6563 foreach(i; 0 .. data.length) { 6564 data[i] = data[i] ^ maskingKey[keyIdx]; 6565 if(keyIdx == 3) 6566 keyIdx = 0; 6567 else 6568 keyIdx++; 6569 } 6570 } 6571 6572 //writeln("SENDING ", headerScratch[0 .. headerScratchPos], data); 6573 llsend(headerScratch[0 .. headerScratchPos]); 6574 llsend(data); 6575 } 6576 6577 static WebSocketFrame read(ref ubyte[] d) { 6578 WebSocketFrame msg; 6579 6580 auto orig = d; 6581 6582 WebSocketFrame needsMoreData() { 6583 d = orig; 6584 return WebSocketFrame.init; 6585 } 6586 6587 if(d.length < 2) 6588 return needsMoreData(); 6589 6590 ubyte b = d[0]; 6591 6592 msg.populated = true; 6593 6594 msg.opcode = cast(WebSocketOpcode) (b & 0x0f); 6595 b >>= 4; 6596 msg.rsv3 = b & 0x01; 6597 b >>= 1; 6598 msg.rsv2 = b & 0x01; 6599 b >>= 1; 6600 msg.rsv1 = b & 0x01; 6601 b >>= 1; 6602 msg.fin = b & 0x01; 6603 6604 b = d[1]; 6605 msg.masked = (b & 0b1000_0000) ? true : false; 6606 msg.lengthIndicator = b & 0b0111_1111; 6607 6608 d = d[2 .. $]; 6609 6610 if(msg.lengthIndicator == 0x7e) { 6611 // 16 bit length 6612 msg.realLength = 0; 6613 6614 if(d.length < 2) return needsMoreData(); 6615 6616 foreach(i; 0 .. 2) { 6617 msg.realLength |= d[0] << ((1-i) * 8); 6618 d = d[1 .. $]; 6619 } 6620 } else if(msg.lengthIndicator == 0x7f) { 6621 // 64 bit length 6622 msg.realLength = 0; 6623 6624 if(d.length < 8) return needsMoreData(); 6625 6626 foreach(i; 0 .. 8) { 6627 msg.realLength |= ulong(d[0]) << ((7-i) * 8); 6628 d = d[1 .. $]; 6629 } 6630 } else { 6631 // 7 bit length 6632 msg.realLength = msg.lengthIndicator; 6633 } 6634 6635 if(msg.masked) { 6636 6637 if(d.length < 4) return needsMoreData(); 6638 6639 msg.maskingKey = d[0 .. 4]; 6640 d = d[4 .. $]; 6641 } 6642 6643 if(msg.realLength > d.length) { 6644 return needsMoreData(); 6645 } 6646 6647 msg.data = d[0 .. cast(size_t) msg.realLength]; 6648 d = d[cast(size_t) msg.realLength .. $]; 6649 6650 return msg; 6651 } 6652 6653 void unmaskInPlace() { 6654 if(this.masked) { 6655 int keyIdx = 0; 6656 foreach(i; 0 .. this.data.length) { 6657 this.data[i] = this.data[i] ^ this.maskingKey[keyIdx]; 6658 if(keyIdx == 3) 6659 keyIdx = 0; 6660 else 6661 keyIdx++; 6662 } 6663 } 6664 } 6665 6666 char[] textData() { 6667 return cast(char[]) data; 6668 } 6669 } 6670 /* } */ 6671 } 6672 6673 6674 version(Windows) 6675 { 6676 version(CRuntime_DigitalMars) 6677 { 6678 extern(C) int setmode(int, int) nothrow @nogc; 6679 } 6680 else version(CRuntime_Microsoft) 6681 { 6682 extern(C) int _setmode(int, int) nothrow @nogc; 6683 alias setmode = _setmode; 6684 } 6685 else static assert(0); 6686 } 6687 6688 version(Posix) { 6689 import core.sys.posix.unistd; 6690 version(CRuntime_Musl) {} else { 6691 private extern(C) int posix_spawn(pid_t*, const char*, void*, void*, const char**, const char**); 6692 } 6693 } 6694 6695 6696 // FIXME: these aren't quite public yet. 6697 //private: 6698 6699 // template for laziness 6700 void startAddonServer()(string arg) { 6701 version(OSX) { 6702 assert(0, "Not implemented"); 6703 } else version(linux) { 6704 import core.sys.posix.unistd; 6705 pid_t pid; 6706 const(char)*[16] args; 6707 args[0] = "ARSD_CGI_ADDON_SERVER"; 6708 args[1] = arg.ptr; 6709 posix_spawn(&pid, "/proc/self/exe", 6710 null, 6711 null, 6712 args.ptr, 6713 null // env 6714 ); 6715 } else version(Windows) { 6716 wchar[2048] filename; 6717 auto len = GetModuleFileNameW(null, filename.ptr, cast(DWORD) filename.length); 6718 if(len == 0 || len == filename.length) 6719 throw new Exception("could not get process name to start helper server"); 6720 6721 STARTUPINFOW startupInfo; 6722 startupInfo.cb = cast(DWORD) startupInfo.sizeof; 6723 PROCESS_INFORMATION processInfo; 6724 6725 import std.utf; 6726 6727 // I *MIGHT* need to run it as a new job or a service... 6728 auto ret = CreateProcessW( 6729 filename.ptr, 6730 toUTF16z(arg), 6731 null, // process attributes 6732 null, // thread attributes 6733 false, // inherit handles 6734 0, // creation flags 6735 null, // environment 6736 null, // working directory 6737 &startupInfo, 6738 &processInfo 6739 ); 6740 6741 if(!ret) 6742 throw new Exception("create process failed"); 6743 6744 // when done with those, if we set them 6745 /* 6746 CloseHandle(hStdInput); 6747 CloseHandle(hStdOutput); 6748 CloseHandle(hStdError); 6749 */ 6750 6751 } else static assert(0, "Websocket server not implemented on this system yet (email me, i can prolly do it if you need it)"); 6752 } 6753 6754 // template for laziness 6755 /* 6756 The websocket server is a single-process, single-thread, event 6757 I/O thing. It is passed websockets from other CGI processes 6758 and is then responsible for handling their messages and responses. 6759 Note that the CGI process is responsible for websocket setup, 6760 including authentication, etc. 6761 6762 It also gets data sent to it by other processes and is responsible 6763 for distributing that, as necessary. 6764 */ 6765 void runWebsocketServer()() { 6766 assert(0, "not implemented"); 6767 } 6768 6769 void sendToWebsocketServer(WebSocket ws, string group) { 6770 assert(0, "not implemented"); 6771 } 6772 6773 void sendToWebsocketServer(string content, string group) { 6774 assert(0, "not implemented"); 6775 } 6776 6777 6778 void runEventServer()() { 6779 runAddonServer("/tmp/arsd_cgi_event_server", new EventSourceServerImplementation()); 6780 } 6781 6782 void runTimerServer()() { 6783 runAddonServer("/tmp/arsd_scheduled_job_server", new ScheduledJobServerImplementation()); 6784 } 6785 6786 version(Posix) { 6787 alias LocalServerConnectionHandle = int; 6788 alias CgiConnectionHandle = int; 6789 alias SocketConnectionHandle = int; 6790 6791 enum INVALID_CGI_CONNECTION_HANDLE = -1; 6792 } else version(Windows) { 6793 alias LocalServerConnectionHandle = HANDLE; 6794 version(embedded_httpd_threads) { 6795 alias CgiConnectionHandle = SOCKET; 6796 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 6797 } else version(fastcgi) { 6798 alias CgiConnectionHandle = void*; // Doesn't actually work! But I don't want compile to fail pointlessly at this point. 6799 enum INVALID_CGI_CONNECTION_HANDLE = null; 6800 } else version(scgi) { 6801 alias CgiConnectionHandle = SOCKET; 6802 enum INVALID_CGI_CONNECTION_HANDLE = INVALID_SOCKET; 6803 } else { /* version(plain_cgi) */ 6804 alias CgiConnectionHandle = HANDLE; 6805 enum INVALID_CGI_CONNECTION_HANDLE = null; 6806 } 6807 alias SocketConnectionHandle = SOCKET; 6808 } 6809 6810 version(with_addon_servers_connections) 6811 LocalServerConnectionHandle openLocalServerConnection()(string name, string arg) { 6812 version(Posix) { 6813 import core.sys.posix.unistd; 6814 import core.sys.posix.sys.un; 6815 6816 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 6817 if(sock == -1) 6818 throw new Exception("socket " ~ to!string(errno)); 6819 6820 scope(failure) 6821 close(sock); 6822 6823 cloexec(sock); 6824 6825 // add-on server processes are assumed to be local, and thus will 6826 // use unix domain sockets. Besides, I want to pass sockets to them, 6827 // so it basically must be local (except for the session server, but meh). 6828 sockaddr_un addr; 6829 addr.sun_family = AF_UNIX; 6830 version(linux) { 6831 // on linux, we will use the abstract namespace 6832 addr.sun_path[0] = 0; 6833 addr.sun_path[1 .. name.length + 1] = cast(typeof(addr.sun_path[])) name[]; 6834 } else { 6835 // but otherwise, just use a file cuz we must. 6836 addr.sun_path[0 .. name.length] = cast(typeof(addr.sun_path[])) name[]; 6837 } 6838 6839 bool alreadyTried; 6840 6841 try_again: 6842 6843 if(connect(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) { 6844 if(!alreadyTried && errno == ECONNREFUSED) { 6845 // try auto-spawning the server, then attempt connection again 6846 startAddonServer(arg); 6847 import core.thread; 6848 Thread.sleep(50.msecs); 6849 alreadyTried = true; 6850 goto try_again; 6851 } else 6852 throw new Exception("connect " ~ to!string(errno)); 6853 } 6854 6855 return sock; 6856 } else version(Windows) { 6857 return null; // FIXME 6858 } 6859 } 6860 6861 version(with_addon_servers_connections) 6862 void closeLocalServerConnection(LocalServerConnectionHandle handle) { 6863 version(Posix) { 6864 import core.sys.posix.unistd; 6865 close(handle); 6866 } else version(Windows) 6867 CloseHandle(handle); 6868 } 6869 6870 void runSessionServer()() { 6871 runAddonServer("/tmp/arsd_session_server", new BasicDataServerImplementation()); 6872 } 6873 6874 version(Posix) 6875 private void makeNonBlocking(int fd) { 6876 import core.sys.posix.fcntl; 6877 auto flags = fcntl(fd, F_GETFL, 0); 6878 if(flags == -1) 6879 throw new Exception("fcntl get"); 6880 flags |= O_NONBLOCK; 6881 auto s = fcntl(fd, F_SETFL, flags); 6882 if(s == -1) 6883 throw new Exception("fcntl set"); 6884 } 6885 6886 import core.stdc.errno; 6887 6888 struct IoOp { 6889 @disable this(); 6890 @disable this(this); 6891 6892 /* 6893 So we want to be able to eventually handle generic sockets too. 6894 */ 6895 6896 enum Read = 1; 6897 enum Write = 2; 6898 enum Accept = 3; 6899 enum ReadSocketHandle = 4; 6900 6901 // Your handler may be called in a different thread than the one that initiated the IO request! 6902 // It is also possible to have multiple io requests being called simultaneously. Use proper thread safety caution. 6903 private bool delegate(IoOp*, int) handler; // returns true if you are done and want it to be closed 6904 private void delegate(IoOp*) closeHandler; 6905 private void delegate(IoOp*) completeHandler; 6906 private int internalFd; 6907 private int operation; 6908 private int bufferLengthAllocated; 6909 private int bufferLengthUsed; 6910 private ubyte[1] internalBuffer; // it can be overallocated! 6911 6912 ubyte[] allocatedBuffer() return { 6913 return internalBuffer.ptr[0 .. bufferLengthAllocated]; 6914 } 6915 6916 ubyte[] usedBuffer() return { 6917 return allocatedBuffer[0 .. bufferLengthUsed]; 6918 } 6919 6920 void reset() { 6921 bufferLengthUsed = 0; 6922 } 6923 6924 int fd() { 6925 return internalFd; 6926 } 6927 } 6928 6929 IoOp* allocateIoOp(int fd, int operation, int bufferSize, bool delegate(IoOp*, int) handler) { 6930 import core.stdc.stdlib; 6931 6932 auto ptr = calloc(IoOp.sizeof + bufferSize, 1); 6933 if(ptr is null) 6934 assert(0); // out of memory! 6935 6936 auto op = cast(IoOp*) ptr; 6937 6938 op.handler = handler; 6939 op.internalFd = fd; 6940 op.operation = operation; 6941 op.bufferLengthAllocated = bufferSize; 6942 op.bufferLengthUsed = 0; 6943 6944 import core.memory; 6945 6946 GC.addRoot(ptr); 6947 6948 return op; 6949 } 6950 6951 void freeIoOp(ref IoOp* ptr) { 6952 6953 import core.memory; 6954 GC.removeRoot(ptr); 6955 6956 import core.stdc.stdlib; 6957 free(ptr); 6958 ptr = null; 6959 } 6960 6961 version(Posix) 6962 version(with_addon_servers_connections) 6963 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 6964 6965 //import std.stdio : writeln; writeln(cast(string) data); 6966 6967 import core.sys.posix.unistd; 6968 6969 auto ret = write(connection, data.ptr, data.length); 6970 if(ret != data.length) { 6971 if(ret == 0 || (ret == -1 && (errno == EPIPE || errno == ETIMEDOUT))) { 6972 // the file is closed, remove it 6973 eis.fileClosed(connection); 6974 } else 6975 throw new Exception("alas " ~ to!string(ret) ~ " " ~ to!string(errno)); // FIXME 6976 } 6977 } 6978 version(Windows) 6979 version(with_addon_servers_connections) 6980 void nonBlockingWrite(EventIoServer eis, int connection, const void[] data) { 6981 // FIXME 6982 } 6983 6984 bool isInvalidHandle(CgiConnectionHandle h) { 6985 return h == INVALID_CGI_CONNECTION_HANDLE; 6986 } 6987 6988 /+ 6989 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsarecv 6990 https://support.microsoft.com/en-gb/help/181611/socket-overlapped-i-o-versus-blocking-nonblocking-mode 6991 https://stackoverflow.com/questions/18018489/should-i-use-iocps-or-overlapped-wsasend-receive 6992 https://docs.microsoft.com/en-us/windows/desktop/fileio/i-o-completion-ports 6993 https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport 6994 https://docs.microsoft.com/en-us/windows/desktop/api/mswsock/nf-mswsock-acceptex 6995 https://docs.microsoft.com/en-us/windows/desktop/Sync/waitable-timer-objects 6996 https://docs.microsoft.com/en-us/windows/desktop/api/synchapi/nf-synchapi-setwaitabletimer 6997 https://docs.microsoft.com/en-us/windows/desktop/Sync/using-a-waitable-timer-with-an-asynchronous-procedure-call 6998 https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsagetoverlappedresult 6999 7000 +/ 7001 7002 /++ 7003 You can customize your server by subclassing the appropriate server. Then, register your 7004 subclass at compile time with the [registerEventIoServer] template, or implement your own 7005 main function and call it yourself. 7006 7007 $(TIP If you make your subclass a `final class`, there is a slight performance improvement.) 7008 +/ 7009 version(with_addon_servers_connections) 7010 interface EventIoServer { 7011 bool handleLocalConnectionData(IoOp* op, int receivedFd); 7012 void handleLocalConnectionClose(IoOp* op); 7013 void handleLocalConnectionComplete(IoOp* op); 7014 void wait_timeout(); 7015 void fileClosed(int fd); 7016 7017 void epoll_fd(int fd); 7018 } 7019 7020 // the sink should buffer it 7021 private void serialize(T)(scope void delegate(scope ubyte[]) sink, T t) { 7022 static if(is(T == struct)) { 7023 foreach(member; __traits(allMembers, T)) 7024 serialize(sink, __traits(getMember, t, member)); 7025 } else static if(is(T : int)) { 7026 // no need to think of endianness just because this is only used 7027 // for local, same-machine stuff anyway. thanks private lol 7028 sink((cast(ubyte*) &t)[0 .. t.sizeof]); 7029 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7030 // these are common enough to optimize 7031 int len = cast(int) t.length; // want length consistent size tho, in case 32 bit program sends to 64 bit server, etc. 7032 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7033 sink(cast(ubyte[]) t[]); 7034 } else static if(is(T : A[], A)) { 7035 // generic array is less optimal but still prolly ok 7036 int len = cast(int) t.length; 7037 sink((cast(ubyte*) &len)[0 .. int.sizeof]); 7038 foreach(item; t) 7039 serialize(sink, item); 7040 } else static assert(0, T.stringof); 7041 } 7042 7043 // all may be stack buffers, so use cautio 7044 private void deserialize(T)(scope ubyte[] delegate(int sz) get, scope void delegate(T) dg) { 7045 static if(is(T == struct)) { 7046 T t; 7047 foreach(member; __traits(allMembers, T)) 7048 deserialize!(typeof(__traits(getMember, T, member)))(get, (mbr) { __traits(getMember, t, member) = mbr; }); 7049 dg(t); 7050 } else static if(is(T : int)) { 7051 // no need to think of endianness just because this is only used 7052 // for local, same-machine stuff anyway. thanks private lol 7053 T t; 7054 auto data = get(t.sizeof); 7055 t = (cast(T[]) data)[0]; 7056 dg(t); 7057 } else static if(is(T == string) || is(T : const(ubyte)[])) { 7058 // these are common enough to optimize 7059 int len; 7060 auto data = get(len.sizeof); 7061 len = (cast(int[]) data)[0]; 7062 7063 /* 7064 typeof(T[0])[2000] stackBuffer; 7065 T buffer; 7066 7067 if(len < stackBuffer.length) 7068 buffer = stackBuffer[0 .. len]; 7069 else 7070 buffer = new T(len); 7071 7072 data = get(len * typeof(T[0]).sizeof); 7073 */ 7074 7075 T t = cast(T) get(len * cast(int) typeof(T.init[0]).sizeof); 7076 7077 dg(t); 7078 } else static if(is(T == E[], E)) { 7079 T t; 7080 int len; 7081 auto data = get(len.sizeof); 7082 len = (cast(int[]) data)[0]; 7083 t.length = len; 7084 foreach(ref e; t) { 7085 deserialize!E(get, (ele) { e = ele; }); 7086 } 7087 dg(t); 7088 } else static assert(0, T.stringof); 7089 } 7090 7091 unittest { 7092 serialize((ubyte[] b) { 7093 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 1); }); 7094 }, 1); 7095 serialize((ubyte[] b) { 7096 deserialize!int( sz => b[0 .. sz], (t) { assert(t == 56674); }); 7097 }, 56674); 7098 ubyte[1000] buffer; 7099 int bufferPoint; 7100 void add(ubyte[] b) { 7101 buffer[bufferPoint .. bufferPoint + b.length] = b[]; 7102 bufferPoint += b.length; 7103 } 7104 ubyte[] get(int sz) { 7105 auto b = buffer[bufferPoint .. bufferPoint + sz]; 7106 bufferPoint += sz; 7107 return b; 7108 } 7109 serialize(&add, "test here"); 7110 bufferPoint = 0; 7111 deserialize!string(&get, (t) { assert(t == "test here"); }); 7112 bufferPoint = 0; 7113 7114 struct Foo { 7115 int a; 7116 ubyte c; 7117 string d; 7118 } 7119 serialize(&add, Foo(403, 37, "amazing")); 7120 bufferPoint = 0; 7121 deserialize!Foo(&get, (t) { 7122 assert(t.a == 403); 7123 assert(t.c == 37); 7124 assert(t.d == "amazing"); 7125 }); 7126 bufferPoint = 0; 7127 } 7128 7129 /* 7130 Here's the way the RPC interface works: 7131 7132 You define the interface that lists the functions you can call on the remote process. 7133 The interface may also have static methods for convenience. These forward to a singleton 7134 instance of an auto-generated class, which actually sends the args over the pipe. 7135 7136 An impl class actually implements it. A receiving server deserializes down the pipe and 7137 calls methods on the class. 7138 7139 I went with the interface to get some nice compiler checking and documentation stuff. 7140 7141 I could have skipped the interface and just implemented it all from the server class definition 7142 itself, but then the usage may call the method instead of rpcing it; I just like having the user 7143 interface and the implementation separate so you aren't tempted to `new impl` to call the methods. 7144 7145 7146 I fiddled with newlines in the mixin string to ensure the assert line numbers matched up to the source code line number. Idk why dmd didn't do this automatically, but it was important to me. 7147 7148 Realistically though the bodies would just be 7149 connection.call(this.mangleof, args...) sooooo. 7150 7151 FIXME: overloads aren't supported 7152 */ 7153 7154 /// Base for storing sessions in an array. Exists primarily for internal purposes and you should generally not use this. 7155 interface SessionObject {} 7156 7157 private immutable void delegate(string[])[string] scheduledJobHandlers; 7158 private immutable void delegate(string[])[string] websocketServers; 7159 7160 version(with_breaking_cgi_features) 7161 mixin(q{ 7162 7163 mixin template ImplementRpcClientInterface(T, string serverPath, string cmdArg) { 7164 static import std.traits; 7165 7166 // derivedMembers on an interface seems to give exactly what I want: the virtual functions we need to implement. so I am just going to use it directly without more filtering. 7167 static foreach(idx, member; __traits(derivedMembers, T)) { 7168 static if(__traits(isVirtualFunction, __traits(getMember, T, member))) 7169 mixin( q{ 7170 std.traits.ReturnType!(__traits(getMember, T, member)) 7171 } ~ member ~ q{(std.traits.Parameters!(__traits(getMember, T, member)) params) 7172 { 7173 SerializationBuffer buffer; 7174 auto i = cast(ushort) idx; 7175 serialize(&buffer.sink, i); 7176 serialize(&buffer.sink, __traits(getMember, T, member).mangleof); 7177 foreach(param; params) 7178 serialize(&buffer.sink, param); 7179 7180 auto sendable = buffer.sendable; 7181 7182 version(Posix) {{ 7183 auto ret = send(connectionHandle, sendable.ptr, sendable.length, 0); 7184 7185 if(ret == -1) { 7186 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7187 } else if(ret == 0) { 7188 throw new Exception("Connection to addon server lost"); 7189 } if(ret < sendable.length) 7190 throw new Exception("Send failed to send all"); 7191 assert(ret == sendable.length); 7192 }} // FIXME Windows impl 7193 7194 static if(!is(typeof(return) == void)) { 7195 // there is a return value; we need to wait for it too 7196 version(Posix) { 7197 ubyte[3000] revBuffer; 7198 auto ret = recv(connectionHandle, revBuffer.ptr, revBuffer.length, 0); 7199 auto got = revBuffer[0 .. ret]; 7200 7201 int dataLocation; 7202 ubyte[] grab(int sz) { 7203 auto dataLocation1 = dataLocation; 7204 dataLocation += sz; 7205 return got[dataLocation1 .. dataLocation]; 7206 } 7207 7208 typeof(return) retu; 7209 deserialize!(typeof(return))(&grab, (a) { retu = a; }); 7210 return retu; 7211 } else { 7212 // FIXME Windows impl 7213 return typeof(return).init; 7214 } 7215 7216 } 7217 }}); 7218 } 7219 7220 private static typeof(this) singletonInstance; 7221 private LocalServerConnectionHandle connectionHandle; 7222 7223 static typeof(this) connection() { 7224 if(singletonInstance is null) { 7225 singletonInstance = new typeof(this)(); 7226 singletonInstance.connect(); 7227 } 7228 return singletonInstance; 7229 } 7230 7231 void connect() { 7232 connectionHandle = openLocalServerConnection(serverPath, cmdArg); 7233 } 7234 7235 void disconnect() { 7236 closeLocalServerConnection(connectionHandle); 7237 } 7238 } 7239 7240 void dispatchRpcServer(Interface, Class)(Class this_, ubyte[] data, int fd) if(is(Class : Interface)) { 7241 ushort calledIdx; 7242 string calledFunction; 7243 7244 int dataLocation; 7245 ubyte[] grab(int sz) { 7246 if(sz == 0) assert(0); 7247 auto d = data[dataLocation .. dataLocation + sz]; 7248 dataLocation += sz; 7249 return d; 7250 } 7251 7252 again: 7253 7254 deserialize!ushort(&grab, (a) { calledIdx = a; }); 7255 deserialize!string(&grab, (a) { calledFunction = a; }); 7256 7257 import std.traits; 7258 7259 sw: switch(calledIdx) { 7260 foreach(idx, memberName; __traits(derivedMembers, Interface)) 7261 static if(__traits(isVirtualFunction, __traits(getMember, Interface, memberName))) { 7262 case idx: 7263 assert(calledFunction == __traits(getMember, Interface, memberName).mangleof); 7264 7265 Parameters!(__traits(getMember, Interface, memberName)) params; 7266 foreach(ref param; params) 7267 deserialize!(typeof(param))(&grab, (a) { param = a; }); 7268 7269 static if(is(ReturnType!(__traits(getMember, Interface, memberName)) == void)) { 7270 __traits(getMember, this_, memberName)(params); 7271 } else { 7272 auto ret = __traits(getMember, this_, memberName)(params); 7273 SerializationBuffer buffer; 7274 serialize(&buffer.sink, ret); 7275 7276 auto sendable = buffer.sendable; 7277 7278 version(Posix) { 7279 auto r = send(fd, sendable.ptr, sendable.length, 0); 7280 if(r == -1) { 7281 throw new Exception("send returned -1, errno: " ~ to!string(errno)); 7282 } else if(r == 0) { 7283 throw new Exception("Connection to addon client lost"); 7284 } if(r < sendable.length) 7285 throw new Exception("Send failed to send all"); 7286 7287 } // FIXME Windows impl 7288 } 7289 break sw; 7290 } 7291 default: assert(0); 7292 } 7293 7294 if(dataLocation != data.length) 7295 goto again; 7296 } 7297 7298 7299 private struct SerializationBuffer { 7300 ubyte[2048] bufferBacking; 7301 int bufferLocation; 7302 void sink(scope ubyte[] data) { 7303 bufferBacking[bufferLocation .. bufferLocation + data.length] = data[]; 7304 bufferLocation += data.length; 7305 } 7306 7307 ubyte[] sendable() return { 7308 return bufferBacking[0 .. bufferLocation]; 7309 } 7310 } 7311 7312 /* 7313 FIXME: 7314 add a version command line arg 7315 version data in the library 7316 management gui as external program 7317 7318 at server with event_fd for each run 7319 use .mangleof in the at function name 7320 7321 i think the at server will have to: 7322 pipe args to the child 7323 collect child output for logging 7324 get child return value for logging 7325 7326 on windows timers work differently. idk how to best combine with the io stuff. 7327 7328 will have to have dump and restore too, so i can restart without losing stuff. 7329 */ 7330 7331 /++ 7332 A convenience object for talking to the [BasicDataServer] from a higher level. 7333 See: [Cgi.getSessionObject]. 7334 7335 You pass it a `Data` struct describing the data you want saved in the session. 7336 Then, this class will generate getter and setter properties that allow access 7337 to that data. 7338 7339 Note that each load and store will be done as-accessed; it doesn't front-load 7340 mutable data nor does it batch updates out of fear of read-modify-write race 7341 conditions. (In fact, right now it does this for everything, but in the future, 7342 I might batch load `immutable` members of the Data struct.) 7343 7344 At some point in the future, I might also let it do different backends, like 7345 a client-side cookie store too, but idk. 7346 7347 Note that the plain-old-data members of your `Data` struct are wrapped by this 7348 interface via a static foreach to make property functions. 7349 7350 See_Also: [MockSession] 7351 +/ 7352 interface Session(Data) : SessionObject { 7353 @property string sessionId() const; 7354 7355 /++ 7356 Starts a new session. Note that a session is also 7357 implicitly started as soon as you write data to it, 7358 so if you need to alter these parameters from their 7359 defaults, be sure to explicitly call this BEFORE doing 7360 any writes to session data. 7361 7362 Params: 7363 idleLifetime = How long, in seconds, the session 7364 should remain in memory when not being read from 7365 or written to. The default is one day. 7366 7367 NOT IMPLEMENTED 7368 7369 useExtendedLifetimeCookie = The session ID is always 7370 stored in a HTTP cookie, and by default, that cookie 7371 is discarded when the user closes their browser. 7372 7373 But if you set this to true, it will use a non-perishable 7374 cookie for the given idleLifetime. 7375 7376 NOT IMPLEMENTED 7377 +/ 7378 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false); 7379 7380 /++ 7381 Regenerates the session ID and updates the associated 7382 cookie. 7383 7384 This is also your chance to change immutable data 7385 (not yet implemented). 7386 +/ 7387 void regenerateId(); 7388 7389 /++ 7390 Terminates this session, deleting all saved data. 7391 +/ 7392 void terminate(); 7393 7394 /++ 7395 Plain-old-data members of your `Data` struct are wrapped here via 7396 the property getters and setters. 7397 7398 If the member is a non-string array, it returns a magical array proxy 7399 object which allows for atomic appends and replaces via overloaded operators. 7400 You can slice this to get a range representing a $(B const) view of the array. 7401 This is to protect you against read-modify-write race conditions. 7402 +/ 7403 static foreach(memberName; __traits(allMembers, Data)) 7404 static if(is(typeof(__traits(getMember, Data, memberName)))) 7405 mixin(q{ 7406 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout; 7407 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value); 7408 }); 7409 7410 } 7411 7412 /++ 7413 An implementation of [Session] that works on real cgi connections utilizing the 7414 [BasicDataServer]. 7415 7416 As opposed to a [MockSession] which is made for testing purposes. 7417 7418 You will not construct one of these directly. See [Cgi.getSessionObject] instead. 7419 +/ 7420 class BasicDataServerSession(Data) : Session!Data { 7421 private Cgi cgi; 7422 private string sessionId_; 7423 7424 public @property string sessionId() const { 7425 return sessionId_; 7426 } 7427 7428 protected @property string sessionId(string s) { 7429 return this.sessionId_ = s; 7430 } 7431 7432 private this(Cgi cgi) { 7433 this.cgi = cgi; 7434 if(auto ptr = "sessionId" in cgi.cookies) 7435 sessionId = (*ptr).length ? *ptr : null; 7436 } 7437 7438 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) { 7439 assert(sessionId is null); 7440 7441 // FIXME: what if there is a session ID cookie, but no corresponding session on the server? 7442 7443 import std.random, std.conv; 7444 sessionId = to!string(uniform(1, long.max)); 7445 7446 BasicDataServer.connection.createSession(sessionId, idleLifetime); 7447 setCookie(); 7448 } 7449 7450 protected void setCookie() { 7451 cgi.setCookie( 7452 "sessionId", sessionId, 7453 0 /* expiration */, 7454 "/" /* path */, 7455 null /* domain */, 7456 true /* http only */, 7457 cgi.https /* if the session is started on https, keep it there, otherwise, be flexible */); 7458 } 7459 7460 void regenerateId() { 7461 if(sessionId is null) { 7462 start(); 7463 return; 7464 } 7465 import std.random, std.conv; 7466 auto oldSessionId = sessionId; 7467 sessionId = to!string(uniform(1, long.max)); 7468 BasicDataServer.connection.renameSession(oldSessionId, sessionId); 7469 setCookie(); 7470 } 7471 7472 void terminate() { 7473 BasicDataServer.connection.destroySession(sessionId); 7474 sessionId = null; 7475 setCookie(); 7476 } 7477 7478 static foreach(memberName; __traits(allMembers, Data)) 7479 static if(is(typeof(__traits(getMember, Data, memberName)))) 7480 mixin(q{ 7481 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7482 if(sessionId is null) 7483 return typeof(return).init; 7484 7485 import std.traits; 7486 auto v = BasicDataServer.connection.getSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName); 7487 if(v.length == 0) 7488 return typeof(return).init; 7489 import std.conv; 7490 // why this cast? to doesn't like being given an inout argument. so need to do it without that, then 7491 // we need to return it and that needed the cast. It should be fine since we basically respect constness.. 7492 // basically. Assuming the session is POD this should be fine. 7493 return cast(typeof(return)) to!(typeof(__traits(getMember, Data, memberName)))(v); 7494 } 7495 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 7496 if(sessionId is null) 7497 start(); 7498 import std.conv; 7499 import std.traits; 7500 BasicDataServer.connection.setSessionData(sessionId, fullyQualifiedName!Data ~ "." ~ memberName, to!string(value)); 7501 return value; 7502 } 7503 }); 7504 } 7505 7506 /++ 7507 A mock object that works like the real session, but doesn't actually interact with any actual database or http connection. 7508 Simply stores the data in its instance members. 7509 +/ 7510 class MockSession(Data) : Session!Data { 7511 pure { 7512 @property string sessionId() const { return "mock"; } 7513 void start(int idleLifetime = 2600 * 24, bool useExtendedLifetimeCookie = false) {} 7514 void regenerateId() {} 7515 void terminate() {} 7516 7517 private Data store_; 7518 7519 static foreach(memberName; __traits(allMembers, Data)) 7520 static if(is(typeof(__traits(getMember, Data, memberName)))) 7521 mixin(q{ 7522 @property inout(typeof(__traits(getMember, Data, memberName))) } ~ memberName ~ q{ () inout { 7523 return __traits(getMember, store_, memberName); 7524 } 7525 @property typeof(__traits(getMember, Data, memberName)) } ~ memberName ~ q{ (typeof(__traits(getMember, Data, memberName)) value) { 7526 return __traits(getMember, store_, memberName) = value; 7527 } 7528 }); 7529 } 7530 } 7531 7532 /++ 7533 Direct interface to the basic data add-on server. You can 7534 typically use [Cgi.getSessionObject] as a more convenient interface. 7535 +/ 7536 version(with_addon_servers_connections) 7537 interface BasicDataServer { 7538 /// 7539 void createSession(string sessionId, int lifetime); 7540 /// 7541 void renewSession(string sessionId, int lifetime); 7542 /// 7543 void destroySession(string sessionId); 7544 /// 7545 void renameSession(string oldSessionId, string newSessionId); 7546 7547 /// 7548 void setSessionData(string sessionId, string dataKey, string dataValue); 7549 /// 7550 string getSessionData(string sessionId, string dataKey); 7551 7552 /// 7553 static BasicDataServerConnection connection() { 7554 return BasicDataServerConnection.connection(); 7555 } 7556 } 7557 7558 version(with_addon_servers_connections) 7559 class BasicDataServerConnection : BasicDataServer { 7560 mixin ImplementRpcClientInterface!(BasicDataServer, "/tmp/arsd_session_server", "--session-server"); 7561 } 7562 7563 version(with_addon_servers) 7564 final class BasicDataServerImplementation : BasicDataServer, EventIoServer { 7565 7566 void createSession(string sessionId, int lifetime) { 7567 sessions[sessionId.idup] = Session(lifetime); 7568 } 7569 void destroySession(string sessionId) { 7570 sessions.remove(sessionId); 7571 } 7572 void renewSession(string sessionId, int lifetime) { 7573 sessions[sessionId].lifetime = lifetime; 7574 } 7575 void renameSession(string oldSessionId, string newSessionId) { 7576 sessions[newSessionId.idup] = sessions[oldSessionId]; 7577 sessions.remove(oldSessionId); 7578 } 7579 void setSessionData(string sessionId, string dataKey, string dataValue) { 7580 if(sessionId !in sessions) 7581 createSession(sessionId, 3600); // FIXME? 7582 sessions[sessionId].values[dataKey.idup] = dataValue.idup; 7583 } 7584 string getSessionData(string sessionId, string dataKey) { 7585 if(auto session = sessionId in sessions) { 7586 if(auto data = dataKey in (*session).values) 7587 return *data; 7588 else 7589 return null; // no such data 7590 7591 } else { 7592 return null; // no session 7593 } 7594 } 7595 7596 7597 protected: 7598 7599 struct Session { 7600 int lifetime; 7601 7602 string[string] values; 7603 } 7604 7605 Session[string] sessions; 7606 7607 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 7608 auto data = op.usedBuffer; 7609 dispatchRpcServer!BasicDataServer(this, data, op.fd); 7610 return false; 7611 } 7612 7613 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 7614 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 7615 void wait_timeout() {} 7616 void fileClosed(int fd) {} // stateless so irrelevant 7617 void epoll_fd(int fd) {} 7618 } 7619 7620 /++ 7621 See [schedule] to make one of these. You then call one of the methods here to set it up: 7622 7623 --- 7624 schedule!fn(args).at(DateTime(2019, 8, 7, 12, 00, 00)); // run the function at August 7, 2019, 12 noon UTC 7625 schedule!fn(args).delay(6.seconds); // run it after waiting 6 seconds 7626 schedule!fn(args).asap(); // run it in the background as soon as the event loop gets around to it 7627 --- 7628 +/ 7629 version(with_addon_servers_connections) 7630 struct ScheduledJobHelper { 7631 private string func; 7632 private string[] args; 7633 private bool consumed; 7634 7635 private this(string func, string[] args) { 7636 this.func = func; 7637 this.args = args; 7638 } 7639 7640 ~this() { 7641 assert(consumed); 7642 } 7643 7644 /++ 7645 Schedules the job to be run at the given time. 7646 +/ 7647 void at(DateTime when, immutable TimeZone timezone = UTC()) { 7648 consumed = true; 7649 7650 auto conn = ScheduledJobServerConnection.connection; 7651 import std.file; 7652 auto st = SysTime(when, timezone); 7653 auto jobId = conn.scheduleJob(1, cast(int) st.toUnixTime(), thisExePath, func, args); 7654 } 7655 7656 /++ 7657 Schedules the job to run at least after the specified delay. 7658 +/ 7659 void delay(Duration delay) { 7660 consumed = true; 7661 7662 auto conn = ScheduledJobServerConnection.connection; 7663 import std.file; 7664 auto jobId = conn.scheduleJob(0, cast(int) delay.total!"seconds", thisExePath, func, args); 7665 } 7666 7667 /++ 7668 Runs the job in the background ASAP. 7669 7670 $(NOTE It may run in a background thread. Don't segfault!) 7671 +/ 7672 void asap() { 7673 consumed = true; 7674 7675 auto conn = ScheduledJobServerConnection.connection; 7676 import std.file; 7677 auto jobId = conn.scheduleJob(0, 1, thisExePath, func, args); 7678 } 7679 7680 /+ 7681 /++ 7682 Schedules the job to recur on the given pattern. 7683 +/ 7684 void recur(string spec) { 7685 7686 } 7687 +/ 7688 } 7689 7690 /++ 7691 First step to schedule a job on the scheduled job server. 7692 7693 The scheduled job needs to be a top-level function that doesn't read any 7694 variables from outside its arguments because it may be run in a new process, 7695 without any context existing later. 7696 7697 You MUST set details on the returned object to actually do anything! 7698 +/ 7699 template schedule(alias fn, T...) if(is(typeof(fn) == function)) { 7700 /// 7701 ScheduledJobHelper schedule(T args) { 7702 // this isn't meant to ever be called, but instead just to 7703 // get the compiler to type check the arguments passed for us 7704 auto sample = delegate() { 7705 fn(args); 7706 }; 7707 string[] sargs; 7708 foreach(arg; args) 7709 sargs ~= to!string(arg); 7710 return ScheduledJobHelper(fn.mangleof, sargs); 7711 } 7712 7713 shared static this() { 7714 scheduledJobHandlers[fn.mangleof] = delegate(string[] sargs) { 7715 import std.traits; 7716 Parameters!fn args; 7717 foreach(idx, ref arg; args) 7718 arg = to!(typeof(arg))(sargs[idx]); 7719 fn(args); 7720 }; 7721 } 7722 } 7723 7724 /// 7725 interface ScheduledJobServer { 7726 /// Use the [schedule] function for a higher-level interface. 7727 int scheduleJob(int whenIs, int when, string executable, string func, string[] args); 7728 /// 7729 void cancelJob(int jobId); 7730 } 7731 7732 version(with_addon_servers_connections) 7733 class ScheduledJobServerConnection : ScheduledJobServer { 7734 mixin ImplementRpcClientInterface!(ScheduledJobServer, "/tmp/arsd_scheduled_job_server", "--timer-server"); 7735 } 7736 7737 version(with_addon_servers) 7738 final class ScheduledJobServerImplementation : ScheduledJobServer, EventIoServer { 7739 // FIXME: we need to handle SIGCHLD in this somehow 7740 // whenIs is 0 for relative, 1 for absolute 7741 protected int scheduleJob(int whenIs, int when, string executable, string func, string[] args) { 7742 auto nj = nextJobId; 7743 nextJobId++; 7744 7745 version(linux) { 7746 import core.sys.linux.timerfd; 7747 import core.sys.linux.epoll; 7748 import core.sys.posix.unistd; 7749 7750 7751 auto fd = timerfd_create(CLOCK_REALTIME, TFD_NONBLOCK | TFD_CLOEXEC); 7752 if(fd == -1) 7753 throw new Exception("fd timer create failed"); 7754 7755 foreach(ref arg; args) 7756 arg = arg.idup; 7757 auto job = Job(executable.idup, func.idup, .dup(args), fd, nj); 7758 7759 itimerspec value; 7760 value.it_value.tv_sec = when; 7761 value.it_value.tv_nsec = 0; 7762 7763 value.it_interval.tv_sec = 0; 7764 value.it_interval.tv_nsec = 0; 7765 7766 if(timerfd_settime(fd, whenIs == 1 ? TFD_TIMER_ABSTIME : 0, &value, null) == -1) 7767 throw new Exception("couldn't set fd timer"); 7768 7769 auto op = allocateIoOp(fd, IoOp.Read, 16, (IoOp* op, int fd) { 7770 jobs.remove(nj); 7771 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, fd, null); 7772 close(fd); 7773 7774 7775 spawnProcess([job.executable, "--timed-job", job.func] ~ job.args); 7776 7777 return true; 7778 }); 7779 scope(failure) 7780 freeIoOp(op); 7781 7782 epoll_event ev; 7783 ev.events = EPOLLIN | EPOLLET; 7784 ev.data.ptr = op; 7785 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, fd, &ev) == -1) 7786 throw new Exception("epoll_ctl " ~ to!string(errno)); 7787 7788 jobs[nj] = job; 7789 return nj; 7790 } else assert(0); 7791 } 7792 7793 protected void cancelJob(int jobId) { 7794 version(linux) { 7795 auto job = jobId in jobs; 7796 if(job is null) 7797 return; 7798 7799 jobs.remove(jobId); 7800 7801 version(linux) { 7802 import core.sys.linux.timerfd; 7803 import core.sys.linux.epoll; 7804 import core.sys.posix.unistd; 7805 epoll_ctl(epoll_fd, EPOLL_CTL_DEL, job.timerfd, null); 7806 close(job.timerfd); 7807 } 7808 } 7809 jobs.remove(jobId); 7810 } 7811 7812 int nextJobId = 1; 7813 static struct Job { 7814 string executable; 7815 string func; 7816 string[] args; 7817 int timerfd; 7818 int id; 7819 } 7820 Job[int] jobs; 7821 7822 7823 // event io server methods below 7824 7825 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 7826 auto data = op.usedBuffer; 7827 dispatchRpcServer!ScheduledJobServer(this, data, op.fd); 7828 return false; 7829 } 7830 7831 void handleLocalConnectionClose(IoOp* op) {} // doesn't really matter, this is a fairly stateless go 7832 void handleLocalConnectionComplete(IoOp* op) {} // again, irrelevant 7833 void wait_timeout() {} 7834 void fileClosed(int fd) {} // stateless so irrelevant 7835 7836 int epoll_fd_; 7837 void epoll_fd(int fd) {this.epoll_fd_ = fd; } 7838 int epoll_fd() { return epoll_fd_; } 7839 } 7840 7841 /// 7842 version(with_addon_servers_connections) 7843 interface EventSourceServer { 7844 /++ 7845 sends this cgi request to the event server so it will be fed events. You should not do anything else with the cgi object after this. 7846 7847 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 7848 7849 See_Also: 7850 [sendEvent] 7851 +/ 7852 public static void adoptConnection(Cgi cgi, in char[] eventUrl) { 7853 /* 7854 If lastEventId is missing or empty, you just get new events as they come. 7855 7856 If it is set from something else, it sends all since then (that are still alive) 7857 down the pipe immediately. 7858 7859 The reason it can come from the header is that's what the standard defines for 7860 browser reconnects. The reason it can come from a query string is just convenience 7861 in catching up in a user-defined manner. 7862 7863 The reason the header overrides the query string is if the browser tries to reconnect, 7864 it will send the header AND the query (it reconnects to the same url), so we just 7865 want to do the restart thing. 7866 7867 Note that if you ask for "0" as the lastEventId, it will get ALL still living events. 7868 */ 7869 string lastEventId = cgi.lastEventId; 7870 if(lastEventId.length == 0 && "lastEventId" in cgi.get) 7871 lastEventId = cgi.get["lastEventId"]; 7872 7873 cgi.setResponseContentType("text/event-stream"); 7874 cgi.write(":\n", false); // to initialize the chunking and send headers before keeping the fd for later 7875 cgi.flush(); 7876 7877 cgi.closed = true; 7878 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 7879 scope(exit) 7880 closeLocalServerConnection(s); 7881 7882 version(fastcgi) 7883 throw new Exception("sending fcgi connections not supported"); 7884 else { 7885 auto fd = cgi.getOutputFileHandle(); 7886 if(isInvalidHandle(fd)) 7887 throw new Exception("bad fd from cgi!"); 7888 7889 EventSourceServerImplementation.SendableEventConnection sec; 7890 sec.populate(cgi.responseChunked, eventUrl, lastEventId); 7891 7892 version(Posix) { 7893 auto res = write_fd(s, cast(void*) &sec, sec.sizeof, fd); 7894 assert(res == sec.sizeof); 7895 } else version(Windows) { 7896 // FIXME 7897 } 7898 } 7899 } 7900 7901 /++ 7902 Sends an event to the event server, starting it if necessary. The event server will distribute it to any listening clients, and store it for `lifetime` seconds for any later listening clients to catch up later. 7903 7904 $(WARNING This API is extremely unstable. I might change it or remove it without notice.) 7905 7906 Params: 7907 url = A string identifying this event "bucket". Listening clients must also connect to this same string. I called it `url` because I envision it being just passed as the url of the request. 7908 event = the event type string, which is used in the Javascript addEventListener API on EventSource 7909 data = the event data. Available in JS as `event.data`. 7910 lifetime = the amount of time to keep this event for replaying on the event server. 7911 7912 See_Also: 7913 [sendEventToEventServer] 7914 +/ 7915 public static void sendEvent(string url, string event, string data, int lifetime) { 7916 auto s = openLocalServerConnection("/tmp/arsd_cgi_event_server", "--event-server"); 7917 scope(exit) 7918 closeLocalServerConnection(s); 7919 7920 EventSourceServerImplementation.SendableEvent sev; 7921 sev.populate(url, event, data, lifetime); 7922 7923 version(Posix) { 7924 auto ret = send(s, &sev, sev.sizeof, 0); 7925 assert(ret == sev.sizeof); 7926 } else version(Windows) { 7927 // FIXME 7928 } 7929 } 7930 7931 /++ 7932 Messages sent to `url` will also be sent to anyone listening on `forwardUrl`. 7933 7934 See_Also: [disconnect] 7935 +/ 7936 void connect(string url, string forwardUrl); 7937 7938 /++ 7939 Disconnects `forwardUrl` from `url` 7940 7941 See_Also: [connect] 7942 +/ 7943 void disconnect(string url, string forwardUrl); 7944 } 7945 7946 /// 7947 version(with_addon_servers) 7948 final class EventSourceServerImplementation : EventSourceServer, EventIoServer { 7949 7950 protected: 7951 7952 void connect(string url, string forwardUrl) { 7953 pipes[url] ~= forwardUrl; 7954 } 7955 void disconnect(string url, string forwardUrl) { 7956 auto t = url in pipes; 7957 if(t is null) 7958 return; 7959 foreach(idx, n; (*t)) 7960 if(n == forwardUrl) { 7961 (*t)[idx] = (*t)[$-1]; 7962 (*t) = (*t)[0 .. $-1]; 7963 break; 7964 } 7965 } 7966 7967 bool handleLocalConnectionData(IoOp* op, int receivedFd) { 7968 if(receivedFd != -1) { 7969 //writeln("GOT FD ", receivedFd, " -- ", op.usedBuffer); 7970 7971 //core.sys.posix.unistd.write(receivedFd, "hello".ptr, 5); 7972 7973 SendableEventConnection* got = cast(SendableEventConnection*) op.usedBuffer.ptr; 7974 7975 auto url = got.url.idup; 7976 eventConnectionsByUrl[url] ~= EventConnection(receivedFd, got.responseChunked > 0 ? true : false); 7977 7978 // FIXME: catch up on past messages here 7979 } else { 7980 auto data = op.usedBuffer; 7981 auto event = cast(SendableEvent*) data.ptr; 7982 7983 if(event.magic == 0xdeadbeef) { 7984 handleInputEvent(event); 7985 7986 if(event.url in pipes) 7987 foreach(pipe; pipes[event.url]) { 7988 event.url = pipe; 7989 handleInputEvent(event); 7990 } 7991 } else { 7992 dispatchRpcServer!EventSourceServer(this, data, op.fd); 7993 } 7994 } 7995 return false; 7996 } 7997 void handleLocalConnectionClose(IoOp* op) { 7998 fileClosed(op.fd); 7999 } 8000 void handleLocalConnectionComplete(IoOp* op) {} 8001 8002 void wait_timeout() { 8003 // just keeping alive 8004 foreach(url, connections; eventConnectionsByUrl) 8005 foreach(connection; connections) 8006 if(connection.needsChunking) 8007 nonBlockingWrite(this, connection.fd, "1b\r\nevent: keepalive\ndata: ok\n\n\r\n"); 8008 else 8009 nonBlockingWrite(this, connection.fd, "event: keepalive\ndata: ok\n\n\r\n"); 8010 } 8011 8012 void fileClosed(int fd) { 8013 outer: foreach(url, ref connections; eventConnectionsByUrl) { 8014 foreach(idx, conn; connections) { 8015 if(fd == conn.fd) { 8016 connections[idx] = connections[$-1]; 8017 connections = connections[0 .. $ - 1]; 8018 continue outer; 8019 } 8020 } 8021 } 8022 } 8023 8024 void epoll_fd(int fd) {} 8025 8026 8027 private: 8028 8029 8030 struct SendableEventConnection { 8031 ubyte responseChunked; 8032 8033 int urlLength; 8034 char[256] urlBuffer = 0; 8035 8036 int lastEventIdLength; 8037 char[32] lastEventIdBuffer = 0; 8038 8039 char[] url() return { 8040 return urlBuffer[0 .. urlLength]; 8041 } 8042 void url(in char[] u) { 8043 urlBuffer[0 .. u.length] = u[]; 8044 urlLength = cast(int) u.length; 8045 } 8046 char[] lastEventId() return { 8047 return lastEventIdBuffer[0 .. lastEventIdLength]; 8048 } 8049 void populate(bool responseChunked, in char[] url, in char[] lastEventId) 8050 in { 8051 assert(url.length < this.urlBuffer.length); 8052 assert(lastEventId.length < this.lastEventIdBuffer.length); 8053 } 8054 do { 8055 this.responseChunked = responseChunked ? 1 : 0; 8056 this.urlLength = cast(int) url.length; 8057 this.lastEventIdLength = cast(int) lastEventId.length; 8058 8059 this.urlBuffer[0 .. url.length] = url[]; 8060 this.lastEventIdBuffer[0 .. lastEventId.length] = lastEventId[]; 8061 } 8062 } 8063 8064 struct SendableEvent { 8065 int magic = 0xdeadbeef; 8066 int urlLength; 8067 char[256] urlBuffer = 0; 8068 int typeLength; 8069 char[32] typeBuffer = 0; 8070 int messageLength; 8071 char[2048 * 4] messageBuffer = 0; // this is an arbitrary limit, it needs to fit comfortably in stack (including in a fiber) and be a single send on the kernel side cuz of the impl... i think this is ok for a unix socket. 8072 int _lifetime; 8073 8074 char[] message() return { 8075 return messageBuffer[0 .. messageLength]; 8076 } 8077 char[] type() return { 8078 return typeBuffer[0 .. typeLength]; 8079 } 8080 char[] url() return { 8081 return urlBuffer[0 .. urlLength]; 8082 } 8083 void url(in char[] u) { 8084 urlBuffer[0 .. u.length] = u[]; 8085 urlLength = cast(int) u.length; 8086 } 8087 int lifetime() { 8088 return _lifetime; 8089 } 8090 8091 /// 8092 void populate(string url, string type, string message, int lifetime) 8093 in { 8094 assert(url.length < this.urlBuffer.length); 8095 assert(type.length < this.typeBuffer.length); 8096 assert(message.length < this.messageBuffer.length); 8097 } 8098 do { 8099 this.urlLength = cast(int) url.length; 8100 this.typeLength = cast(int) type.length; 8101 this.messageLength = cast(int) message.length; 8102 this._lifetime = lifetime; 8103 8104 this.urlBuffer[0 .. url.length] = url[]; 8105 this.typeBuffer[0 .. type.length] = type[]; 8106 this.messageBuffer[0 .. message.length] = message[]; 8107 } 8108 } 8109 8110 struct EventConnection { 8111 int fd; 8112 bool needsChunking; 8113 } 8114 8115 private EventConnection[][string] eventConnectionsByUrl; 8116 private string[][string] pipes; 8117 8118 private void handleInputEvent(scope SendableEvent* event) { 8119 static int eventId; 8120 8121 static struct StoredEvent { 8122 int id; 8123 string type; 8124 string message; 8125 int lifetimeRemaining; 8126 } 8127 8128 StoredEvent[][string] byUrl; 8129 8130 int thisId = ++eventId; 8131 8132 if(event.lifetime) 8133 byUrl[event.url.idup] ~= StoredEvent(thisId, event.type.idup, event.message.idup, event.lifetime); 8134 8135 auto connectionsPtr = event.url in eventConnectionsByUrl; 8136 EventConnection[] connections; 8137 if(connectionsPtr is null) 8138 return; 8139 else 8140 connections = *connectionsPtr; 8141 8142 char[4096] buffer; 8143 char[] formattedMessage; 8144 8145 void append(const char[] a) { 8146 // the 6's here are to leave room for a HTTP chunk header, if it proves necessary 8147 buffer[6 + formattedMessage.length .. 6 + formattedMessage.length + a.length] = a[]; 8148 formattedMessage = buffer[6 .. 6 + formattedMessage.length + a.length]; 8149 } 8150 8151 import std.algorithm.iteration; 8152 8153 if(connections.length) { 8154 append("id: "); 8155 append(to!string(thisId)); 8156 append("\n"); 8157 8158 append("event: "); 8159 append(event.type); 8160 append("\n"); 8161 8162 foreach(line; event.message.splitter("\n")) { 8163 append("data: "); 8164 append(line); 8165 append("\n"); 8166 } 8167 8168 append("\n"); 8169 } 8170 8171 // chunk it for HTTP! 8172 auto len = toHex(formattedMessage.length); 8173 buffer[4 .. 6] = "\r\n"[]; 8174 buffer[4 - len.length .. 4] = len[]; 8175 buffer[6 + formattedMessage.length] = '\r'; 8176 buffer[6 + formattedMessage.length + 1] = '\n'; 8177 8178 auto chunkedMessage = buffer[4 - len.length .. 6 + formattedMessage.length +2]; 8179 // done 8180 8181 // FIXME: send back requests when needed 8182 // FIXME: send a single ":\n" every 15 seconds to keep alive 8183 8184 foreach(connection; connections) { 8185 if(connection.needsChunking) { 8186 nonBlockingWrite(this, connection.fd, chunkedMessage); 8187 } else { 8188 nonBlockingWrite(this, connection.fd, formattedMessage); 8189 } 8190 } 8191 } 8192 } 8193 8194 void runAddonServer(EIS)(string localListenerName, EIS eis) if(is(EIS : EventIoServer)) { 8195 version(Posix) { 8196 8197 import core.sys.posix.unistd; 8198 import core.sys.posix.fcntl; 8199 import core.sys.posix.sys.un; 8200 8201 import core.sys.posix.signal; 8202 signal(SIGPIPE, SIG_IGN); 8203 8204 static extern(C) void sigchldhandler(int) { 8205 int status; 8206 import w = core.sys.posix.sys.wait; 8207 w.wait(&status); 8208 } 8209 signal(SIGCHLD, &sigchldhandler); 8210 8211 int sock = socket(AF_UNIX, SOCK_STREAM, 0); 8212 if(sock == -1) 8213 throw new Exception("socket " ~ to!string(errno)); 8214 8215 scope(failure) 8216 close(sock); 8217 8218 cloexec(sock); 8219 8220 // add-on server processes are assumed to be local, and thus will 8221 // use unix domain sockets. Besides, I want to pass sockets to them, 8222 // so it basically must be local (except for the session server, but meh). 8223 sockaddr_un addr; 8224 addr.sun_family = AF_UNIX; 8225 version(linux) { 8226 // on linux, we will use the abstract namespace 8227 addr.sun_path[0] = 0; 8228 addr.sun_path[1 .. localListenerName.length + 1] = cast(typeof(addr.sun_path[])) localListenerName[]; 8229 } else { 8230 // but otherwise, just use a file cuz we must. 8231 addr.sun_path[0 .. localListenerName.length] = cast(typeof(addr.sun_path[])) localListenerName[]; 8232 } 8233 8234 if(bind(sock, cast(sockaddr*) &addr, addr.sizeof) == -1) 8235 throw new Exception("bind " ~ to!string(errno)); 8236 8237 if(listen(sock, 128) == -1) 8238 throw new Exception("listen " ~ to!string(errno)); 8239 8240 makeNonBlocking(sock); 8241 8242 version(linux) { 8243 import core.sys.linux.epoll; 8244 auto epoll_fd = epoll_create1(EPOLL_CLOEXEC); 8245 if(epoll_fd == -1) 8246 throw new Exception("epoll_create1 " ~ to!string(errno)); 8247 scope(failure) 8248 close(epoll_fd); 8249 } else { 8250 import core.sys.posix.poll; 8251 } 8252 8253 version(linux) 8254 eis.epoll_fd = epoll_fd; 8255 8256 auto acceptOp = allocateIoOp(sock, IoOp.Read, 0, null); 8257 scope(exit) 8258 freeIoOp(acceptOp); 8259 8260 version(linux) { 8261 epoll_event ev; 8262 ev.events = EPOLLIN | EPOLLET; 8263 ev.data.ptr = acceptOp; 8264 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, sock, &ev) == -1) 8265 throw new Exception("epoll_ctl " ~ to!string(errno)); 8266 8267 epoll_event[64] events; 8268 } else { 8269 pollfd[] pollfds; 8270 IoOp*[int] ioops; 8271 pollfds ~= pollfd(sock, POLLIN); 8272 ioops[sock] = acceptOp; 8273 } 8274 8275 import core.time : MonoTime, seconds; 8276 8277 MonoTime timeout = MonoTime.currTime + 15.seconds; 8278 8279 while(true) { 8280 8281 // FIXME: it should actually do a timerfd that runs on any thing that hasn't been run recently 8282 8283 int timeout_milliseconds = 0; // -1; // infinite 8284 8285 timeout_milliseconds = cast(int) (timeout - MonoTime.currTime).total!"msecs"; 8286 if(timeout_milliseconds < 0) 8287 timeout_milliseconds = 0; 8288 8289 //writeln("waiting for ", name); 8290 8291 version(linux) { 8292 auto nfds = epoll_wait(epoll_fd, events.ptr, events.length, timeout_milliseconds); 8293 if(nfds == -1) { 8294 if(errno == EINTR) 8295 continue; 8296 throw new Exception("epoll_wait " ~ to!string(errno)); 8297 } 8298 } else { 8299 int nfds = poll(pollfds.ptr, cast(int) pollfds.length, timeout_milliseconds); 8300 size_t lastIdx = 0; 8301 } 8302 8303 if(nfds == 0) { 8304 eis.wait_timeout(); 8305 timeout += 15.seconds; 8306 } 8307 8308 foreach(idx; 0 .. nfds) { 8309 version(linux) { 8310 auto flags = events[idx].events; 8311 auto ioop = cast(IoOp*) events[idx].data.ptr; 8312 } else { 8313 IoOp* ioop; 8314 foreach(tidx, thing; pollfds[lastIdx .. $]) { 8315 if(thing.revents) { 8316 ioop = ioops[thing.fd]; 8317 lastIdx += tidx + 1; 8318 break; 8319 } 8320 } 8321 } 8322 8323 //writeln(flags, " ", ioop.fd); 8324 8325 void newConnection() { 8326 // on edge triggering, it is important that we get it all 8327 while(true) { 8328 version(Android) { 8329 auto size = cast(int) addr.sizeof; 8330 } else { 8331 auto size = cast(uint) addr.sizeof; 8332 } 8333 auto ns = accept(sock, cast(sockaddr*) &addr, &size); 8334 if(ns == -1) { 8335 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8336 // all done, got it all 8337 break; 8338 } 8339 throw new Exception("accept " ~ to!string(errno)); 8340 } 8341 cloexec(ns); 8342 8343 makeNonBlocking(ns); 8344 auto niop = allocateIoOp(ns, IoOp.ReadSocketHandle, 4096 * 4, &eis.handleLocalConnectionData); 8345 niop.closeHandler = &eis.handleLocalConnectionClose; 8346 niop.completeHandler = &eis.handleLocalConnectionComplete; 8347 scope(failure) freeIoOp(niop); 8348 8349 version(linux) { 8350 epoll_event nev; 8351 nev.events = EPOLLIN | EPOLLET; 8352 nev.data.ptr = niop; 8353 if(epoll_ctl(epoll_fd, EPOLL_CTL_ADD, ns, &nev) == -1) 8354 throw new Exception("epoll_ctl " ~ to!string(errno)); 8355 } else { 8356 bool found = false; 8357 foreach(ref pfd; pollfds) { 8358 if(pfd.fd < 0) { 8359 pfd.fd = ns; 8360 found = true; 8361 } 8362 } 8363 if(!found) 8364 pollfds ~= pollfd(ns, POLLIN); 8365 ioops[ns] = niop; 8366 } 8367 } 8368 } 8369 8370 bool newConnectionCondition() { 8371 version(linux) 8372 return ioop.fd == sock && (flags & EPOLLIN); 8373 else 8374 return pollfds[idx].fd == sock && (pollfds[idx].revents & POLLIN); 8375 } 8376 8377 if(newConnectionCondition()) { 8378 newConnection(); 8379 } else if(ioop.operation == IoOp.ReadSocketHandle) { 8380 while(true) { 8381 int in_fd; 8382 auto got = read_fd(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length, &in_fd); 8383 if(got == -1) { 8384 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8385 // all done, got it all 8386 if(ioop.completeHandler) 8387 ioop.completeHandler(ioop); 8388 break; 8389 } 8390 throw new Exception("recv " ~ to!string(errno)); 8391 } 8392 8393 if(got == 0) { 8394 if(ioop.closeHandler) { 8395 ioop.closeHandler(ioop); 8396 version(linux) {} // nothing needed 8397 else { 8398 foreach(ref pfd; pollfds) { 8399 if(pfd.fd == ioop.fd) 8400 pfd.fd = -1; 8401 } 8402 } 8403 } 8404 close(ioop.fd); 8405 freeIoOp(ioop); 8406 break; 8407 } 8408 8409 ioop.bufferLengthUsed = cast(int) got; 8410 ioop.handler(ioop, in_fd); 8411 } 8412 } else if(ioop.operation == IoOp.Read) { 8413 while(true) { 8414 auto got = read(ioop.fd, ioop.allocatedBuffer.ptr, ioop.allocatedBuffer.length); 8415 if(got == -1) { 8416 if(errno == EAGAIN || errno == EWOULDBLOCK) { 8417 // all done, got it all 8418 if(ioop.completeHandler) 8419 ioop.completeHandler(ioop); 8420 break; 8421 } 8422 throw new Exception("recv " ~ to!string(ioop.fd) ~ " errno " ~ to!string(errno)); 8423 } 8424 8425 if(got == 0) { 8426 if(ioop.closeHandler) 8427 ioop.closeHandler(ioop); 8428 close(ioop.fd); 8429 freeIoOp(ioop); 8430 break; 8431 } 8432 8433 ioop.bufferLengthUsed = cast(int) got; 8434 if(ioop.handler(ioop, ioop.fd)) { 8435 close(ioop.fd); 8436 freeIoOp(ioop); 8437 break; 8438 } 8439 } 8440 } 8441 8442 // EPOLLHUP? 8443 } 8444 } 8445 } else version(Windows) { 8446 8447 // set up a named pipe 8448 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms724251(v=vs.85).aspx 8449 // https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-wsaduplicatesocketw 8450 // https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-getnamedpipeserverprocessid 8451 8452 } else static assert(0); 8453 } 8454 8455 8456 version(with_sendfd) 8457 // copied from the web and ported from C 8458 // see https://stackoverflow.com/questions/2358684/can-i-share-a-file-descriptor-to-another-process-on-linux-or-are-they-local-to-t 8459 ssize_t write_fd(int fd, void *ptr, size_t nbytes, int sendfd) { 8460 msghdr msg; 8461 iovec[1] iov; 8462 8463 version(OSX) { 8464 //msg.msg_accrights = cast(cattr_t) &sendfd; 8465 //msg.msg_accrightslen = int.sizeof; 8466 } else version(Android) { 8467 } else { 8468 union ControlUnion { 8469 cmsghdr cm; 8470 char[CMSG_SPACE(int.sizeof)] control; 8471 } 8472 8473 ControlUnion control_un; 8474 cmsghdr* cmptr; 8475 8476 msg.msg_control = control_un.control.ptr; 8477 msg.msg_controllen = control_un.control.length; 8478 8479 cmptr = CMSG_FIRSTHDR(&msg); 8480 cmptr.cmsg_len = CMSG_LEN(int.sizeof); 8481 cmptr.cmsg_level = SOL_SOCKET; 8482 cmptr.cmsg_type = SCM_RIGHTS; 8483 *(cast(int *) CMSG_DATA(cmptr)) = sendfd; 8484 } 8485 8486 msg.msg_name = null; 8487 msg.msg_namelen = 0; 8488 8489 iov[0].iov_base = ptr; 8490 iov[0].iov_len = nbytes; 8491 msg.msg_iov = iov.ptr; 8492 msg.msg_iovlen = 1; 8493 8494 return sendmsg(fd, &msg, 0); 8495 } 8496 8497 version(with_sendfd) 8498 // copied from the web and ported from C 8499 ssize_t read_fd(int fd, void *ptr, size_t nbytes, int *recvfd) { 8500 msghdr msg; 8501 iovec[1] iov; 8502 ssize_t n; 8503 int newfd; 8504 8505 version(OSX) { 8506 //msg.msg_accrights = cast(cattr_t) recvfd; 8507 //msg.msg_accrightslen = int.sizeof; 8508 } else version(Android) { 8509 } else { 8510 union ControlUnion { 8511 cmsghdr cm; 8512 char[CMSG_SPACE(int.sizeof)] control; 8513 } 8514 ControlUnion control_un; 8515 cmsghdr* cmptr; 8516 8517 msg.msg_control = control_un.control.ptr; 8518 msg.msg_controllen = control_un.control.length; 8519 } 8520 8521 msg.msg_name = null; 8522 msg.msg_namelen = 0; 8523 8524 iov[0].iov_base = ptr; 8525 iov[0].iov_len = nbytes; 8526 msg.msg_iov = iov.ptr; 8527 msg.msg_iovlen = 1; 8528 8529 if ( (n = recvmsg(fd, &msg, 0)) <= 0) 8530 return n; 8531 8532 version(OSX) { 8533 //if(msg.msg_accrightslen != int.sizeof) 8534 //*recvfd = -1; 8535 } else version(Android) { 8536 } else { 8537 if ( (cmptr = CMSG_FIRSTHDR(&msg)) != null && 8538 cmptr.cmsg_len == CMSG_LEN(int.sizeof)) { 8539 if (cmptr.cmsg_level != SOL_SOCKET) 8540 throw new Exception("control level != SOL_SOCKET"); 8541 if (cmptr.cmsg_type != SCM_RIGHTS) 8542 throw new Exception("control type != SCM_RIGHTS"); 8543 *recvfd = *(cast(int *) CMSG_DATA(cmptr)); 8544 } else 8545 *recvfd = -1; /* descriptor was not passed */ 8546 } 8547 8548 return n; 8549 } 8550 /* end read_fd */ 8551 8552 8553 /* 8554 Event source stuff 8555 8556 The api is: 8557 8558 sendEvent(string url, string type, string data, int timeout = 60*10); 8559 8560 attachEventListener(string url, int fd, lastId) 8561 8562 8563 It just sends to all attached listeners, and stores it until the timeout 8564 for replaying via lastEventId. 8565 */ 8566 8567 /* 8568 Session process stuff 8569 8570 it stores it all. the cgi object has a session object that can grab it 8571 8572 session may be done in the same process if possible, there is a version 8573 switch to choose if you want to override. 8574 */ 8575 8576 struct DispatcherDefinition(alias dispatchHandler, DispatcherDetails = typeof(null)) {// if(is(typeof(dispatchHandler("str", Cgi.init, void) == bool))) { // bool delegate(string urlPrefix, Cgi cgi) dispatchHandler; 8577 alias handler = dispatchHandler; 8578 string urlPrefix; 8579 bool rejectFurther; 8580 immutable(DispatcherDetails) details; 8581 } 8582 8583 private string urlify(string name) pure { 8584 return beautify(name, '-', true); 8585 } 8586 8587 private string beautify(string name, char space = ' ', bool allLowerCase = false) pure { 8588 if(name == "id") 8589 return allLowerCase ? name : "ID"; 8590 8591 char[160] buffer; 8592 int bufferIndex = 0; 8593 bool shouldCap = true; 8594 bool shouldSpace; 8595 bool lastWasCap; 8596 foreach(idx, char ch; name) { 8597 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 8598 8599 if((ch >= 'A' && ch <= 'Z') || ch == '_') { 8600 if(lastWasCap) { 8601 // two caps in a row, don't change. Prolly acronym. 8602 } else { 8603 if(idx) 8604 shouldSpace = true; // new word, add space 8605 } 8606 8607 lastWasCap = true; 8608 } else { 8609 lastWasCap = false; 8610 } 8611 8612 if(shouldSpace) { 8613 buffer[bufferIndex++] = space; 8614 if(bufferIndex == buffer.length) return name; // out of space, just give up, not that important 8615 shouldSpace = false; 8616 } 8617 if(shouldCap) { 8618 if(ch >= 'a' && ch <= 'z') 8619 ch -= 32; 8620 shouldCap = false; 8621 } 8622 if(allLowerCase && ch >= 'A' && ch <= 'Z') 8623 ch += 32; 8624 buffer[bufferIndex++] = ch; 8625 } 8626 return buffer[0 .. bufferIndex].idup; 8627 } 8628 8629 /* 8630 string urlFor(alias func)() { 8631 return __traits(identifier, func); 8632 } 8633 */ 8634 8635 /++ 8636 UDA: The name displayed to the user in auto-generated HTML. 8637 8638 Default is `beautify(identifier)`. 8639 +/ 8640 struct DisplayName { 8641 string name; 8642 } 8643 8644 /++ 8645 UDA: The name used in the URL or web parameter. 8646 8647 Default is `urlify(identifier)` for functions and `identifier` for parameters and data members. 8648 +/ 8649 struct UrlName { 8650 string name; 8651 } 8652 8653 /++ 8654 UDA: default format to respond for this method 8655 +/ 8656 struct DefaultFormat { string value; } 8657 8658 class MissingArgumentException : Exception { 8659 string functionName; 8660 string argumentName; 8661 string argumentType; 8662 8663 this(string functionName, string argumentName, string argumentType, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 8664 this.functionName = functionName; 8665 this.argumentName = argumentName; 8666 this.argumentType = argumentType; 8667 8668 super("Missing Argument: " ~ this.argumentName, file, line, next); 8669 } 8670 } 8671 8672 /++ 8673 You can throw this from an api handler to indicate a 404 response. This is done by the presentExceptionAsHtml function in the presenter. 8674 8675 History: 8676 Added December 15, 2021 (dub v10.5) 8677 +/ 8678 class ResourceNotFoundException : Exception { 8679 string resourceType; 8680 string resourceId; 8681 8682 this(string resourceType, string resourceId, string file = __FILE__, size_t line = __LINE__, Throwable next = null) { 8683 this.resourceType = resourceType; 8684 this.resourceId = resourceId; 8685 8686 super("Resource not found: " ~ resourceType ~ " " ~ resourceId, file, line, next); 8687 } 8688 8689 } 8690 8691 /++ 8692 This can be attached to any constructor or function called from the cgi system. 8693 8694 If it is present, the function argument can NOT be set from web params, but instead 8695 is set to the return value of the given `func`. 8696 8697 If `func` can take a parameter of type [Cgi], it will be passed the one representing 8698 the current request. Otherwise, it must take zero arguments. 8699 8700 Any params in your function of type `Cgi` are automatically assumed to take the cgi object 8701 for the connection. Any of type [Session] (with an argument) is also assumed to come from 8702 the cgi object. 8703 8704 const arguments are also supported. 8705 +/ 8706 struct ifCalledFromWeb(alias func) {} 8707 8708 // it only looks at query params for GET requests, the rest must be in the body for a function argument. 8709 auto callFromCgi(alias method, T)(T dg, Cgi cgi) { 8710 8711 // FIXME: any array of structs should also be settable or gettable from csv as well. 8712 8713 // FIXME: think more about checkboxes and bools. 8714 8715 import std.traits; 8716 8717 Parameters!method params; 8718 alias idents = ParameterIdentifierTuple!method; 8719 alias defaults = ParameterDefaults!method; 8720 8721 const(string)[] names; 8722 const(string)[] values; 8723 8724 // first, check for missing arguments and initialize to defaults if necessary 8725 8726 static if(is(typeof(method) P == __parameters)) 8727 foreach(idx, param; P) {{ 8728 // see: mustNotBeSetFromWebParams 8729 static if(is(param : Cgi)) { 8730 static assert(!is(param == immutable)); 8731 cast() params[idx] = cgi; 8732 } else static if(is(param == Session!D, D)) { 8733 static assert(!is(param == immutable)); 8734 cast() params[idx] = cgi.getSessionObject!D(); 8735 } else { 8736 bool populated; 8737 foreach(uda; __traits(getAttributes, P[idx .. idx + 1])) { 8738 static if(is(uda == ifCalledFromWeb!func, alias func)) { 8739 static if(is(typeof(func(cgi)))) 8740 params[idx] = func(cgi); 8741 else 8742 params[idx] = func(); 8743 8744 populated = true; 8745 } 8746 } 8747 8748 if(!populated) { 8749 static if(__traits(compiles, { params[idx] = param.getAutomaticallyForCgi(cgi); } )) { 8750 params[idx] = param.getAutomaticallyForCgi(cgi); 8751 populated = true; 8752 } 8753 } 8754 8755 if(!populated) { 8756 auto ident = idents[idx]; 8757 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 8758 if(ident !in cgi.get) { 8759 static if(is(defaults[idx] == void)) { 8760 static if(is(param == bool)) 8761 params[idx] = false; 8762 else 8763 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 8764 } else 8765 params[idx] = defaults[idx]; 8766 } 8767 } else { 8768 if(ident !in cgi.post) { 8769 static if(is(defaults[idx] == void)) { 8770 static if(is(param == bool)) 8771 params[idx] = false; 8772 else 8773 throw new MissingArgumentException(__traits(identifier, method), ident, param.stringof); 8774 } else 8775 params[idx] = defaults[idx]; 8776 } 8777 } 8778 } 8779 } 8780 }} 8781 8782 // second, parse the arguments in order to build up arrays, etc. 8783 8784 static bool setVariable(T)(string name, string paramName, T* what, string value) { 8785 static if(is(T == struct)) { 8786 if(name == paramName) { 8787 *what = T.init; 8788 return true; 8789 } else { 8790 // could be a child. gonna allow either obj.field OR obj[field] 8791 8792 string afterName; 8793 8794 if(name[paramName.length] == '[') { 8795 int count = 1; 8796 auto idx = paramName.length + 1; 8797 while(idx < name.length && count > 0) { 8798 if(name[idx] == '[') 8799 count++; 8800 else if(name[idx] == ']') { 8801 count--; 8802 if(count == 0) break; 8803 } 8804 idx++; 8805 } 8806 8807 if(idx == name.length) 8808 return false; // malformed 8809 8810 auto insideBrackets = name[paramName.length + 1 .. idx]; 8811 afterName = name[idx + 1 .. $]; 8812 8813 name = name[0 .. paramName.length]; 8814 8815 paramName = insideBrackets; 8816 8817 } else if(name[paramName.length] == '.') { 8818 paramName = name[paramName.length + 1 .. $]; 8819 name = paramName; 8820 int p = 0; 8821 foreach(ch; paramName) { 8822 if(ch == '.' || ch == '[') 8823 break; 8824 p++; 8825 } 8826 8827 afterName = paramName[p .. $]; 8828 paramName = paramName[0 .. p]; 8829 } else { 8830 return false; 8831 } 8832 8833 if(paramName.length) 8834 // set the child member 8835 switch(paramName) { 8836 foreach(idx, memberName; __traits(allMembers, T)) 8837 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 8838 // data member! 8839 case memberName: 8840 return setVariable(name ~ afterName, paramName, &(__traits(getMember, *what, memberName)), value); 8841 } 8842 default: 8843 // ok, not a member 8844 } 8845 } 8846 8847 return false; 8848 } else static if(is(T == enum)) { 8849 *what = to!T(value); 8850 return true; 8851 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 8852 *what = to!T(value); 8853 return true; 8854 } else static if(is(T == bool)) { 8855 *what = value == "1" || value == "yes" || value == "t" || value == "true" || value == "on"; 8856 return true; 8857 } else static if(is(T == K[], K)) { 8858 K tmp; 8859 if(name == paramName) { 8860 // direct - set and append 8861 if(setVariable(name, paramName, &tmp, value)) { 8862 (*what) ~= tmp; 8863 return true; 8864 } else { 8865 return false; 8866 } 8867 } else { 8868 // child, append to last element 8869 // FIXME: what about range violations??? 8870 auto ptr = &(*what)[(*what).length - 1]; 8871 return setVariable(name, paramName, ptr, value); 8872 8873 } 8874 } else static if(is(T == V[K], K, V)) { 8875 // assoc array, name[key] is valid 8876 if(name == paramName) { 8877 // no action necessary 8878 return true; 8879 } else if(name[paramName.length] == '[') { 8880 int count = 1; 8881 auto idx = paramName.length + 1; 8882 while(idx < name.length && count > 0) { 8883 if(name[idx] == '[') 8884 count++; 8885 else if(name[idx] == ']') { 8886 count--; 8887 if(count == 0) break; 8888 } 8889 idx++; 8890 } 8891 if(idx == name.length) 8892 return false; // malformed 8893 8894 auto insideBrackets = name[paramName.length + 1 .. idx]; 8895 auto afterName = name[idx + 1 .. $]; 8896 8897 auto k = to!K(insideBrackets); 8898 V v; 8899 if(auto ptr = k in *what) 8900 v = *ptr; 8901 8902 name = name[0 .. paramName.length]; 8903 //writeln(name, afterName, " ", paramName); 8904 8905 auto ret = setVariable(name ~ afterName, paramName, &v, value); 8906 if(ret) { 8907 (*what)[k] = v; 8908 return true; 8909 } 8910 } 8911 8912 return false; 8913 } else { 8914 static assert(0, "unsupported type for cgi call " ~ T.stringof); 8915 } 8916 8917 //return false; 8918 } 8919 8920 void setArgument(string name, string value) { 8921 int p; 8922 foreach(ch; name) { 8923 if(ch == '.' || ch == '[') 8924 break; 8925 p++; 8926 } 8927 8928 auto paramName = name[0 .. p]; 8929 8930 sw: switch(paramName) { 8931 static if(is(typeof(method) P == __parameters)) 8932 foreach(idx, param; P) { 8933 static if(mustNotBeSetFromWebParams!(P[idx], __traits(getAttributes, P[idx .. idx + 1]))) { 8934 // cannot be set from the outside 8935 } else { 8936 case idents[idx]: 8937 static if(is(param == Cgi.UploadedFile)) { 8938 params[idx] = cgi.files[name]; 8939 } else { 8940 setVariable(name, paramName, ¶ms[idx], value); 8941 } 8942 break sw; 8943 } 8944 } 8945 default: 8946 // ignore; not relevant argument 8947 } 8948 } 8949 8950 if(cgi.requestMethod == Cgi.RequestMethod.GET) { 8951 names = cgi.allGetNamesInOrder; 8952 values = cgi.allGetValuesInOrder; 8953 } else { 8954 names = cgi.allPostNamesInOrder; 8955 values = cgi.allPostValuesInOrder; 8956 } 8957 8958 foreach(idx, name; names) { 8959 setArgument(name, values[idx]); 8960 } 8961 8962 static if(is(ReturnType!method == void)) { 8963 typeof(null) ret; 8964 dg(params); 8965 } else { 8966 auto ret = dg(params); 8967 } 8968 8969 // FIXME: format return values 8970 // options are: json, html, csv. 8971 // also may need to wrap in envelope format: none, html, or json. 8972 return ret; 8973 } 8974 8975 private bool mustNotBeSetFromWebParams(T, attrs...)() { 8976 static if(is(T : const(Cgi))) { 8977 return true; 8978 } else static if(is(T : const(Session!D), D)) { 8979 return true; 8980 } else static if(__traits(compiles, T.getAutomaticallyForCgi(Cgi.init))) { 8981 return true; 8982 } else { 8983 foreach(uda; attrs) 8984 static if(is(uda == ifCalledFromWeb!func, alias func)) 8985 return true; 8986 return false; 8987 } 8988 } 8989 8990 private bool hasIfCalledFromWeb(attrs...)() { 8991 foreach(uda; attrs) 8992 static if(is(uda == ifCalledFromWeb!func, alias func)) 8993 return true; 8994 return false; 8995 } 8996 8997 /++ 8998 Implies POST path for the thing itself, then GET will get the automatic form. 8999 9000 The given customizer, if present, will be called as a filter on the Form object. 9001 9002 History: 9003 Added December 27, 2020 9004 +/ 9005 template AutomaticForm(alias customizer) { } 9006 9007 /++ 9008 This is meant to be returned by a function that takes a form POST submission. You 9009 want to set the url of the new resource it created, which is set as the http 9010 Location header for a "201 Created" result, and you can also set a separate 9011 destination for browser users, which it sets via a "Refresh" header. 9012 9013 The `resourceRepresentation` should generally be the thing you just created, and 9014 it will be the body of the http response when formatted through the presenter. 9015 The exact thing is up to you - it could just return an id, or the whole object, or 9016 perhaps a partial object. 9017 9018 Examples: 9019 --- 9020 class Test : WebObject { 9021 @(Cgi.RequestMethod.POST) 9022 CreatedResource!int makeThing(string value) { 9023 return CreatedResource!int(value.to!int, "/resources/id"); 9024 } 9025 } 9026 --- 9027 9028 History: 9029 Added December 18, 2021 9030 +/ 9031 struct CreatedResource(T) { 9032 static if(!is(T == void)) 9033 T resourceRepresentation; 9034 string resourceUrl; 9035 string refreshUrl; 9036 } 9037 9038 /+ 9039 /++ 9040 This can be attached as a UDA to a handler to add a http Refresh header on a 9041 successful run. (It will not be attached if the function throws an exception.) 9042 This will refresh the browser the given number of seconds after the page loads, 9043 to the url returned by `urlFunc`, which can be either a static function or a 9044 member method of the current handler object. 9045 9046 You might use this for a POST handler that is normally used from ajax, but you 9047 want it to degrade gracefully to a temporarily flashed message before reloading 9048 the main page. 9049 9050 History: 9051 Added December 18, 2021 9052 +/ 9053 struct Refresh(alias urlFunc) { 9054 int waitInSeconds; 9055 9056 string url() { 9057 static if(__traits(isStaticFunction, urlFunc)) 9058 return urlFunc(); 9059 else static if(is(urlFunc : string)) 9060 return urlFunc; 9061 } 9062 } 9063 +/ 9064 9065 /+ 9066 /++ 9067 Sets a filter to be run before 9068 9069 A before function can do validations of params and log and stop the function from running. 9070 +/ 9071 template Before(alias b) {} 9072 template After(alias b) {} 9073 +/ 9074 9075 /+ 9076 Argument conversions: for the most part, it is to!Thing(string). 9077 9078 But arrays and structs are a bit different. Arrays come from the cgi array. Thus 9079 they are passed 9080 9081 arr=foo&arr=bar <-- notice the same name. 9082 9083 Structs are first declared with an empty thing, then have their members set individually, 9084 with dot notation. The members are not required, just the initial declaration. 9085 9086 struct Foo { 9087 int a; 9088 string b; 9089 } 9090 void test(Foo foo){} 9091 9092 foo&foo.a=5&foo.b=str <-- the first foo declares the arg, the others set the members 9093 9094 Arrays of structs use this declaration. 9095 9096 void test(Foo[] foo) {} 9097 9098 foo&foo.a=5&foo.b=bar&foo&foo.a=9 9099 9100 You can use a hidden input field in HTML forms to achieve this. The value of the naked name 9101 declaration is ignored. 9102 9103 Mind that order matters! The declaration MUST come first in the string. 9104 9105 Arrays of struct members follow this rule recursively. 9106 9107 struct Foo { 9108 int[] a; 9109 } 9110 9111 foo&foo.a=1&foo.a=2&foo&foo.a=1 9112 9113 9114 Associative arrays are formatted with brackets, after a declaration, like structs: 9115 9116 foo&foo[key]=value&foo[other_key]=value 9117 9118 9119 Note: for maximum compatibility with outside code, keep your types simple. Some libraries 9120 do not support the strict ordering requirements to work with these struct protocols. 9121 9122 FIXME: also perhaps accept application/json to better work with outside trash. 9123 9124 9125 Return values are also auto-formatted according to user-requested type: 9126 for json, it loops over and converts. 9127 for html, basic types are strings. Arrays are <ol>. Structs are <dl>. Arrays of structs are tables! 9128 +/ 9129 9130 /++ 9131 A web presenter is responsible for rendering things to HTML to be usable 9132 in a web browser. 9133 9134 They are passed as template arguments to the base classes of [WebObject] 9135 9136 Responsible for displaying stuff as HTML. You can put this into your own aggregate 9137 and override it. Use forwarding and specialization to customize it. 9138 9139 When you inherit from it, pass your own class as the CRTP argument. This lets the base 9140 class templates and your overridden templates work with each other. 9141 9142 --- 9143 class MyPresenter : WebPresenter!(MyPresenter) { 9144 @Override 9145 void presentSuccessfulReturnAsHtml(T : CustomType)(Cgi cgi, T ret, typeof(null) meta) { 9146 // present the CustomType 9147 } 9148 @Override 9149 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9150 // handle everything else via the super class, which will call 9151 // back to your class when appropriate 9152 super.presentSuccessfulReturnAsHtml(cgi, ret); 9153 } 9154 } 9155 --- 9156 9157 The meta argument in there can be overridden by your own facility. 9158 9159 +/ 9160 class WebPresenter(CRTP) { 9161 9162 /// A UDA version of the built-in `override`, to be used for static template polymorphism 9163 /// If you override a plain method, use `override`. If a template, use `@Override`. 9164 enum Override; 9165 9166 string script() { 9167 return ` 9168 `; 9169 } 9170 9171 string style() { 9172 return ` 9173 :root { 9174 --mild-border: #ccc; 9175 --middle-border: #999; 9176 --accent-color: #f2f2f2; 9177 --sidebar-color: #fefefe; 9178 } 9179 ` ~ genericFormStyling() ~ genericSiteStyling(); 9180 } 9181 9182 string genericFormStyling() { 9183 return 9184 q"css 9185 table.automatic-data-display { 9186 border-collapse: collapse; 9187 border: solid 1px var(--mild-border); 9188 } 9189 9190 table.automatic-data-display td { 9191 vertical-align: top; 9192 border: solid 1px var(--mild-border); 9193 padding: 2px 4px; 9194 } 9195 9196 table.automatic-data-display th { 9197 border: solid 1px var(--mild-border); 9198 border-bottom: solid 1px var(--middle-border); 9199 padding: 2px 4px; 9200 } 9201 9202 ol.automatic-data-display { 9203 margin: 0px; 9204 list-style-position: inside; 9205 padding: 0px; 9206 } 9207 9208 dl.automatic-data-display { 9209 9210 } 9211 9212 .automatic-form { 9213 max-width: 600px; 9214 } 9215 9216 .form-field { 9217 margin: 0.5em; 9218 padding-left: 0.5em; 9219 } 9220 9221 .label-text { 9222 display: block; 9223 font-weight: bold; 9224 margin-left: -0.5em; 9225 } 9226 9227 .submit-button-holder { 9228 padding-left: 2em; 9229 } 9230 9231 .add-array-button { 9232 9233 } 9234 css"; 9235 } 9236 9237 string genericSiteStyling() { 9238 return 9239 q"css 9240 * { box-sizing: border-box; } 9241 html, body { margin: 0px; } 9242 body { 9243 font-family: sans-serif; 9244 } 9245 header { 9246 background: var(--accent-color); 9247 height: 64px; 9248 } 9249 footer { 9250 background: var(--accent-color); 9251 height: 64px; 9252 } 9253 #site-container { 9254 display: flex; 9255 } 9256 main { 9257 flex: 1 1 auto; 9258 order: 2; 9259 min-height: calc(100vh - 64px - 64px); 9260 padding: 4px; 9261 padding-left: 1em; 9262 } 9263 #sidebar { 9264 flex: 0 0 16em; 9265 order: 1; 9266 background: var(--sidebar-color); 9267 } 9268 css"; 9269 } 9270 9271 import arsd.dom; 9272 Element htmlContainer() { 9273 auto document = new Document(q"html 9274 <!DOCTYPE html> 9275 <html class="no-script"> 9276 <head> 9277 <script>document.documentElement.classList.remove("no-script");</script> 9278 <style>.no-script requires-script { display: none; }</style> 9279 <title>D Application</title> 9280 <link rel="stylesheet" href="style.css" /> 9281 </head> 9282 <body> 9283 <header></header> 9284 <div id="site-container"> 9285 <main></main> 9286 <div id="sidebar"></div> 9287 </div> 9288 <footer></footer> 9289 <script src="script.js"></script> 9290 </body> 9291 </html> 9292 html", true, true); 9293 9294 return document.requireSelector("main"); 9295 } 9296 9297 /// Renders a response as an HTTP error 9298 void renderBasicError(Cgi cgi, int httpErrorCode) { 9299 cgi.setResponseStatus(getHttpCodeText(httpErrorCode)); 9300 auto c = htmlContainer(); 9301 c.innerText = getHttpCodeText(httpErrorCode); 9302 cgi.setResponseContentType("text/html; charset=utf-8"); 9303 cgi.write(c.parentDocument.toString(), true); 9304 } 9305 9306 template methodMeta(alias method) { 9307 enum methodMeta = null; 9308 } 9309 9310 void presentSuccessfulReturn(T, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9311 // FIXME? format? 9312 (cast(CRTP) this).presentSuccessfulReturnAsHtml(cgi, ret, meta); 9313 } 9314 9315 /// typeof(null) (which is also used to represent functions returning `void`) do nothing 9316 /// in the default presenter - allowing the function to have full low-level control over the 9317 /// response. 9318 void presentSuccessfulReturn(T : typeof(null), Meta)(Cgi cgi, T ret, Meta meta, string format) { 9319 // nothing intentionally! 9320 } 9321 9322 /// Redirections are forwarded to [Cgi.setResponseLocation] 9323 void presentSuccessfulReturn(T : Redirection, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9324 cgi.setResponseLocation(ret.to, true, getHttpCodeText(ret.code)); 9325 } 9326 9327 /// [CreatedResource]s send code 201 and will set the given urls, then present the given representation. 9328 void presentSuccessfulReturn(T : CreatedResource!R, Meta, R)(Cgi cgi, T ret, Meta meta, string format) { 9329 cgi.setResponseStatus(getHttpCodeText(201)); 9330 if(ret.resourceUrl.length) 9331 cgi.header("Location: " ~ ret.resourceUrl); 9332 if(ret.refreshUrl.length) 9333 cgi.header("Refresh: 0;" ~ ret.refreshUrl); 9334 static if(!is(R == void)) 9335 presentSuccessfulReturn(cgi, ret.resourceRepresentation, meta, format); 9336 } 9337 9338 /// Multiple responses deconstruct the algebraic type and forward to the appropriate handler at runtime 9339 void presentSuccessfulReturn(T : MultipleResponses!Types, Meta, Types...)(Cgi cgi, T ret, Meta meta, string format) { 9340 bool outputted = false; 9341 foreach(index, type; Types) { 9342 if(ret.contains == index) { 9343 assert(!outputted); 9344 outputted = true; 9345 (cast(CRTP) this).presentSuccessfulReturn(cgi, ret.payload[index], meta, format); 9346 } 9347 } 9348 if(!outputted) 9349 assert(0); 9350 } 9351 9352 /// An instance of the [arsd.dom.FileResource] interface has its own content type; assume it is a download of some sort. 9353 void presentSuccessfulReturn(T : FileResource, Meta)(Cgi cgi, T ret, Meta meta, string format) { 9354 cgi.setCache(true); // not necessarily true but meh 9355 cgi.setResponseContentType(ret.contentType); 9356 cgi.write(ret.getData(), true); 9357 } 9358 9359 /// And the default handler for HTML will call [formatReturnValueAsHtml] and place it inside the [htmlContainer]. 9360 void presentSuccessfulReturnAsHtml(T)(Cgi cgi, T ret, typeof(null) meta) { 9361 auto container = this.htmlContainer(); 9362 container.appendChild(formatReturnValueAsHtml(ret)); 9363 cgi.write(container.parentDocument.toString(), true); 9364 } 9365 9366 /++ 9367 If you override this, you will need to cast the exception type `t` dynamically, 9368 but can then use the template arguments here to refer back to the function. 9369 9370 `func` is an alias to the method itself, and `dg` is a callable delegate to the same 9371 method on the live object. You could, in theory, change arguments and retry, but I 9372 provide that information mostly with the expectation that you will use them to make 9373 useful forms or richer error messages for the user. 9374 +/ 9375 void presentExceptionAsHtml(alias func, T)(Cgi cgi, Throwable t, T dg) { 9376 Form af; 9377 foreach(attr; __traits(getAttributes, func)) { 9378 static if(__traits(isSame, attr, AutomaticForm)) { 9379 af = createAutomaticFormForFunction!(func)(dg); 9380 } 9381 } 9382 presentExceptionAsHtmlImpl(cgi, t, af); 9383 } 9384 9385 void presentExceptionAsHtmlImpl(Cgi cgi, Throwable t, Form automaticForm) { 9386 if(auto e = cast(ResourceNotFoundException) t) { 9387 auto container = this.htmlContainer(); 9388 9389 container.addChild("p", e.msg); 9390 9391 if(!cgi.outputtedResponseData) 9392 cgi.setResponseStatus("404 Not Found"); 9393 cgi.write(container.parentDocument.toString(), true); 9394 } else if(auto mae = cast(MissingArgumentException) t) { 9395 if(automaticForm is null) 9396 goto generic; 9397 auto container = this.htmlContainer(); 9398 if(cgi.requestMethod == Cgi.RequestMethod.POST) 9399 container.appendChild(Element.make("p", "Argument `" ~ mae.argumentName ~ "` of type `" ~ mae.argumentType ~ "` is missing")); 9400 container.appendChild(automaticForm); 9401 9402 cgi.write(container.parentDocument.toString(), true); 9403 } else { 9404 generic: 9405 auto container = this.htmlContainer(); 9406 9407 // import std.stdio; writeln(t.toString()); 9408 9409 container.appendChild(exceptionToElement(t)); 9410 9411 container.addChild("h4", "GET"); 9412 foreach(k, v; cgi.get) { 9413 auto deets = container.addChild("details"); 9414 deets.addChild("summary", k); 9415 deets.addChild("div", v); 9416 } 9417 9418 container.addChild("h4", "POST"); 9419 foreach(k, v; cgi.post) { 9420 auto deets = container.addChild("details"); 9421 deets.addChild("summary", k); 9422 deets.addChild("div", v); 9423 } 9424 9425 9426 if(!cgi.outputtedResponseData) 9427 cgi.setResponseStatus("500 Internal Server Error"); 9428 cgi.write(container.parentDocument.toString(), true); 9429 } 9430 } 9431 9432 Element exceptionToElement(Throwable t) { 9433 auto div = Element.make("div"); 9434 div.addClass("exception-display"); 9435 9436 div.addChild("p", t.msg); 9437 div.addChild("p", "Inner code origin: " ~ typeid(t).name ~ "@" ~ t.file ~ ":" ~ to!string(t.line)); 9438 9439 auto pre = div.addChild("pre"); 9440 string s; 9441 s = t.toString(); 9442 Element currentBox; 9443 bool on = false; 9444 foreach(line; s.splitLines) { 9445 if(!on && line.startsWith("-----")) 9446 on = true; 9447 if(!on) continue; 9448 if(line.indexOf("arsd/") != -1) { 9449 if(currentBox is null) { 9450 currentBox = pre.addChild("details"); 9451 currentBox.addChild("summary", "Framework code"); 9452 } 9453 currentBox.addChild("span", line ~ "\n"); 9454 } else { 9455 pre.addChild("span", line ~ "\n"); 9456 currentBox = null; 9457 } 9458 } 9459 9460 return div; 9461 } 9462 9463 /++ 9464 Returns an element for a particular type 9465 +/ 9466 Element elementFor(T)(string displayName, string name, Element function() udaSuggestion) { 9467 import std.traits; 9468 9469 auto div = Element.make("div"); 9470 div.addClass("form-field"); 9471 9472 static if(is(T == Cgi.UploadedFile)) { 9473 Element lbl; 9474 if(displayName !is null) { 9475 lbl = div.addChild("label"); 9476 lbl.addChild("span", displayName, "label-text"); 9477 lbl.appendText(" "); 9478 } else { 9479 lbl = div; 9480 } 9481 auto i = lbl.addChild("input", name); 9482 i.attrs.name = name; 9483 i.attrs.type = "file"; 9484 } else static if(is(T == enum)) { 9485 Element lbl; 9486 if(displayName !is null) { 9487 lbl = div.addChild("label"); 9488 lbl.addChild("span", displayName, "label-text"); 9489 lbl.appendText(" "); 9490 } else { 9491 lbl = div; 9492 } 9493 auto i = lbl.addChild("select", name); 9494 i.attrs.name = name; 9495 9496 foreach(memberName; __traits(allMembers, T)) 9497 i.addChild("option", memberName); 9498 9499 } else static if(is(T == struct)) { 9500 if(displayName !is null) 9501 div.addChild("span", displayName, "label-text"); 9502 auto fieldset = div.addChild("fieldset"); 9503 fieldset.addChild("legend", beautify(T.stringof)); // FIXME 9504 fieldset.addChild("input", name); 9505 foreach(idx, memberName; __traits(allMembers, T)) 9506 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9507 fieldset.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(beautify(memberName), name ~ "." ~ memberName, null /* FIXME: pull off the UDA */)); 9508 } 9509 } else static if(isSomeString!T || isIntegral!T || isFloatingPoint!T) { 9510 Element lbl; 9511 if(displayName !is null) { 9512 lbl = div.addChild("label"); 9513 lbl.addChild("span", displayName, "label-text"); 9514 lbl.appendText(" "); 9515 } else { 9516 lbl = div; 9517 } 9518 Element i; 9519 if(udaSuggestion) { 9520 i = udaSuggestion(); 9521 lbl.appendChild(i); 9522 } else { 9523 i = lbl.addChild("input", name); 9524 } 9525 i.attrs.name = name; 9526 static if(isSomeString!T) 9527 i.attrs.type = "text"; 9528 else 9529 i.attrs.type = "number"; 9530 if(i.tagName == "textarea") 9531 i.textContent = to!string(T.init); 9532 else 9533 i.attrs.value = to!string(T.init); 9534 } else static if(is(T == bool)) { 9535 Element lbl; 9536 if(displayName !is null) { 9537 lbl = div.addChild("label"); 9538 lbl.addChild("span", displayName, "label-text"); 9539 lbl.appendText(" "); 9540 } else { 9541 lbl = div; 9542 } 9543 auto i = lbl.addChild("input", name); 9544 i.attrs.type = "checkbox"; 9545 i.attrs.value = "true"; 9546 i.attrs.name = name; 9547 } else static if(is(T == K[], K)) { 9548 auto templ = div.addChild("template"); 9549 templ.appendChild(elementFor!(K)(null, name, null /* uda??*/)); 9550 if(displayName !is null) 9551 div.addChild("span", displayName, "label-text"); 9552 auto btn = div.addChild("button"); 9553 btn.addClass("add-array-button"); 9554 btn.attrs.type = "button"; 9555 btn.innerText = "Add"; 9556 btn.attrs.onclick = q{ 9557 var a = document.importNode(this.parentNode.firstChild.content, true); 9558 this.parentNode.insertBefore(a, this); 9559 }; 9560 } else static if(is(T == V[K], K, V)) { 9561 div.innerText = "assoc array not implemented for automatic form at this time"; 9562 } else { 9563 static assert(0, "unsupported type for cgi call " ~ T.stringof); 9564 } 9565 9566 9567 return div; 9568 } 9569 9570 /// creates a form for gathering the function's arguments 9571 Form createAutomaticFormForFunction(alias method, T)(T dg) { 9572 9573 auto form = cast(Form) Element.make("form"); 9574 9575 form.method = "POST"; // FIXME 9576 9577 form.addClass("automatic-form"); 9578 9579 string formDisplayName = beautify(__traits(identifier, method)); 9580 foreach(attr; __traits(getAttributes, method)) 9581 static if(is(typeof(attr) == DisplayName)) 9582 formDisplayName = attr.name; 9583 form.addChild("h3", formDisplayName); 9584 9585 import std.traits; 9586 9587 //Parameters!method params; 9588 //alias idents = ParameterIdentifierTuple!method; 9589 //alias defaults = ParameterDefaults!method; 9590 9591 static if(is(typeof(method) P == __parameters)) 9592 foreach(idx, _; P) {{ 9593 9594 alias param = P[idx .. idx + 1]; 9595 9596 static if(!mustNotBeSetFromWebParams!(param[0], __traits(getAttributes, param))) { 9597 string displayName = beautify(__traits(identifier, param)); 9598 Element function() element; 9599 foreach(attr; __traits(getAttributes, param)) { 9600 static if(is(typeof(attr) == DisplayName)) 9601 displayName = attr.name; 9602 else static if(is(typeof(attr) : typeof(element))) { 9603 element = attr; 9604 } 9605 } 9606 auto i = form.appendChild(elementFor!(param)(displayName, __traits(identifier, param), element)); 9607 if(i.querySelector("input[type=file]") !is null) 9608 form.setAttribute("enctype", "multipart/form-data"); 9609 } 9610 }} 9611 9612 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 9613 9614 return form; 9615 } 9616 9617 /// creates a form for gathering object members (for the REST object thing right now) 9618 Form createAutomaticFormForObject(T)(T obj) { 9619 auto form = cast(Form) Element.make("form"); 9620 9621 form.addClass("automatic-form"); 9622 9623 form.addChild("h3", beautify(__traits(identifier, T))); 9624 9625 import std.traits; 9626 9627 //Parameters!method params; 9628 //alias idents = ParameterIdentifierTuple!method; 9629 //alias defaults = ParameterDefaults!method; 9630 9631 foreach(idx, memberName; __traits(derivedMembers, T)) {{ 9632 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 9633 string displayName = beautify(memberName); 9634 Element function() element; 9635 foreach(attr; __traits(getAttributes, __traits(getMember, T, memberName))) 9636 static if(is(typeof(attr) == DisplayName)) 9637 displayName = attr.name; 9638 else static if(is(typeof(attr) : typeof(element))) 9639 element = attr; 9640 form.appendChild(elementFor!(typeof(__traits(getMember, T, memberName)))(displayName, memberName, element)); 9641 9642 form.setValue(memberName, to!string(__traits(getMember, obj, memberName))); 9643 }}} 9644 9645 form.addChild("div", Html(`<input type="submit" value="Submit" />`), "submit-button-holder"); 9646 9647 return form; 9648 } 9649 9650 /// 9651 Element formatReturnValueAsHtml(T)(T t) { 9652 import std.traits; 9653 9654 static if(is(T == typeof(null))) { 9655 return Element.make("span"); 9656 } else static if(is(T : Element)) { 9657 return t; 9658 } else static if(is(T == MultipleResponses!Types, Types...)) { 9659 foreach(index, type; Types) { 9660 if(t.contains == index) 9661 return formatReturnValueAsHtml(t.payload[index]); 9662 } 9663 assert(0); 9664 } else static if(is(T == Paginated!E, E)) { 9665 auto e = Element.make("div").addClass("paginated-result"); 9666 e.appendChild(formatReturnValueAsHtml(t.items)); 9667 if(t.nextPageUrl.length) 9668 e.appendChild(Element.make("a", "Next Page", t.nextPageUrl)); 9669 return e; 9670 } else static if(isIntegral!T || isSomeString!T || isFloatingPoint!T) { 9671 return Element.make("span", to!string(t), "automatic-data-display"); 9672 } else static if(is(T == V[K], K, V)) { 9673 auto dl = Element.make("dl"); 9674 dl.addClass("automatic-data-display associative-array"); 9675 foreach(k, v; t) { 9676 dl.addChild("dt", to!string(k)); 9677 dl.addChild("dd", formatReturnValueAsHtml(v)); 9678 } 9679 return dl; 9680 } else static if(is(T == struct)) { 9681 auto dl = Element.make("dl"); 9682 dl.addClass("automatic-data-display struct"); 9683 9684 foreach(idx, memberName; __traits(allMembers, T)) 9685 static if(__traits(compiles, __traits(getMember, T, memberName).offsetof)) { 9686 dl.addChild("dt", beautify(memberName)); 9687 dl.addChild("dd", formatReturnValueAsHtml(__traits(getMember, t, memberName))); 9688 } 9689 9690 return dl; 9691 } else static if(is(T == bool)) { 9692 return Element.make("span", t ? "true" : "false", "automatic-data-display"); 9693 } else static if(is(T == E[], E)) { 9694 static if(is(E : RestObject!Proxy, Proxy)) { 9695 // treat RestObject similar to struct 9696 auto table = cast(Table) Element.make("table"); 9697 table.addClass("automatic-data-display"); 9698 string[] names; 9699 foreach(idx, memberName; __traits(derivedMembers, E)) 9700 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 9701 names ~= beautify(memberName); 9702 } 9703 table.appendHeaderRow(names); 9704 9705 foreach(l; t) { 9706 auto tr = table.appendRow(); 9707 foreach(idx, memberName; __traits(derivedMembers, E)) 9708 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 9709 static if(memberName == "id") { 9710 string val = to!string(__traits(getMember, l, memberName)); 9711 tr.addChild("td", Element.make("a", val, E.stringof.toLower ~ "s/" ~ val)); // FIXME 9712 } else { 9713 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 9714 } 9715 } 9716 } 9717 9718 return table; 9719 } else static if(is(E == struct)) { 9720 // an array of structs is kinda special in that I like 9721 // having those formatted as tables. 9722 auto table = cast(Table) Element.make("table"); 9723 table.addClass("automatic-data-display"); 9724 string[] names; 9725 foreach(idx, memberName; __traits(allMembers, E)) 9726 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 9727 names ~= beautify(memberName); 9728 } 9729 table.appendHeaderRow(names); 9730 9731 foreach(l; t) { 9732 auto tr = table.appendRow(); 9733 foreach(idx, memberName; __traits(allMembers, E)) 9734 static if(__traits(compiles, __traits(getMember, E, memberName).offsetof)) { 9735 tr.addChild("td", formatReturnValueAsHtml(__traits(getMember, l, memberName))); 9736 } 9737 } 9738 9739 return table; 9740 } else { 9741 // otherwise, I will just make a list. 9742 auto ol = Element.make("ol"); 9743 ol.addClass("automatic-data-display"); 9744 foreach(e; t) 9745 ol.addChild("li", formatReturnValueAsHtml(e)); 9746 return ol; 9747 } 9748 } else static if(is(T : Object)) { 9749 static if(is(typeof(t.toHtml()))) // FIXME: maybe i will make this an interface 9750 return Element.make("div", t.toHtml()); 9751 else 9752 return Element.make("div", t.toString()); 9753 } else static assert(0, "bad return value for cgi call " ~ T.stringof); 9754 9755 assert(0); 9756 } 9757 9758 } 9759 9760 /++ 9761 The base class for the [dispatcher] function and object support. 9762 +/ 9763 class WebObject { 9764 //protected Cgi cgi; 9765 9766 protected void initialize(Cgi cgi) { 9767 //this.cgi = cgi; 9768 } 9769 } 9770 9771 /++ 9772 Can return one of the given types, decided at runtime. The syntax 9773 is to declare all the possible types in the return value, then you 9774 can `return typeof(return)(...value...)` to construct it. 9775 9776 It has an auto-generated constructor for each value it can hold. 9777 9778 --- 9779 MultipleResponses!(Redirection, string) getData(int how) { 9780 if(how & 1) 9781 return typeof(return)(Redirection("http://dpldocs.info/")); 9782 else 9783 return typeof(return)("hi there!"); 9784 } 9785 --- 9786 9787 If you have lots of returns, you could, inside the function, `alias r = typeof(return);` to shorten it a little. 9788 +/ 9789 struct MultipleResponses(T...) { 9790 private size_t contains; 9791 private union { 9792 private T payload; 9793 } 9794 9795 static foreach(index, type; T) 9796 public this(type t) { 9797 contains = index; 9798 payload[index] = t; 9799 } 9800 9801 /++ 9802 This is primarily for testing. It is your way of getting to the response. 9803 9804 Let's say you wanted to test that one holding a Redirection and a string actually 9805 holds a string, by name of "test": 9806 9807 --- 9808 auto valueToTest = your_test_function(); 9809 9810 valueToTest.visit( 9811 (Redirection r) { assert(0); }, // got a redirection instead of a string, fail the test 9812 (string s) { assert(s == "test"); } // right value, go ahead and test it. 9813 ); 9814 --- 9815 9816 History: 9817 Was horribly broken until June 16, 2022. Ironically, I wrote it for tests but never actually tested it. 9818 It tried to use alias lambdas before, but runtime delegates work much better so I changed it. 9819 +/ 9820 void visit(Handlers...)(Handlers handlers) { 9821 template findHandler(type, int count, HandlersToCheck...) { 9822 static if(HandlersToCheck.length == 0) 9823 enum findHandler = -1; 9824 else { 9825 static if(is(typeof(HandlersToCheck[0].init(type.init)))) 9826 enum findHandler = count; 9827 else 9828 enum findHandler = findHandler!(type, count + 1, HandlersToCheck[1 .. $]); 9829 } 9830 } 9831 foreach(index, type; T) { 9832 enum handlerIndex = findHandler!(type, 0, Handlers); 9833 static if(handlerIndex == -1) 9834 static assert(0, "Type " ~ type.stringof ~ " was not handled by visitor"); 9835 else { 9836 if(index == this.contains) 9837 handlers[handlerIndex](this.payload[index]); 9838 } 9839 } 9840 } 9841 9842 /+ 9843 auto toArsdJsvar()() { 9844 import arsd.jsvar; 9845 return var(null); 9846 } 9847 +/ 9848 } 9849 9850 // FIXME: implement this somewhere maybe 9851 struct RawResponse { 9852 int code; 9853 string[] headers; 9854 const(ubyte)[] responseBody; 9855 } 9856 9857 /++ 9858 You can return this from [WebObject] subclasses for redirections. 9859 9860 (though note the static types means that class must ALWAYS redirect if 9861 you return this directly. You might want to return [MultipleResponses] if it 9862 can be conditional) 9863 +/ 9864 struct Redirection { 9865 string to; /// The URL to redirect to. 9866 int code = 303; /// The HTTP code to return. 9867 } 9868 9869 /++ 9870 Serves a class' methods, as a kind of low-state RPC over the web. To be used with [dispatcher]. 9871 9872 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar] unless you have overriden 9873 the presenter in the dispatcher. 9874 9875 FIXME: explain this better 9876 9877 You can overload functions to a limited extent: you can provide a zero-arg and non-zero-arg function, 9878 and non-zero-arg functions can filter via UDAs for various http methods. Do not attempt other overloads, 9879 the runtime result of that is undefined. 9880 9881 A method is assumed to allow any http method unless it lists some in UDAs, in which case it is limited to only those. 9882 (this might change, like maybe i will use pure as an indicator GET is ok. idk.) 9883 9884 $(WARNING 9885 --- 9886 // legal in D, undefined runtime behavior with cgi.d, it may call either method 9887 // even if you put different URL udas on it, the current code ignores them. 9888 void foo(int a) {} 9889 void foo(string a) {} 9890 --- 9891 ) 9892 9893 See_Also: [serveRestObject], [serveStaticFile] 9894 +/ 9895 auto serveApi(T)(string urlPrefix) { 9896 assert(urlPrefix[$ - 1] == '/'); 9897 return serveApiInternal!T(urlPrefix); 9898 } 9899 9900 private string nextPieceFromSlash(ref string remainingUrl) { 9901 if(remainingUrl.length == 0) 9902 return remainingUrl; 9903 int slash = 0; 9904 while(slash < remainingUrl.length && remainingUrl[slash] != '/') // && remainingUrl[slash] != '.') 9905 slash++; 9906 9907 // I am specifically passing `null` to differentiate it vs empty string 9908 // so in your ctor, `items` means new T(null) and `items/` means new T("") 9909 auto ident = remainingUrl.length == 0 ? null : remainingUrl[0 .. slash]; 9910 // so if it is the last item, the dot can be used to load an alternative view 9911 // otherwise tho the dot is considered part of the identifier 9912 // FIXME 9913 9914 // again notice "" vs null here! 9915 if(slash == remainingUrl.length) 9916 remainingUrl = null; 9917 else 9918 remainingUrl = remainingUrl[slash + 1 .. $]; 9919 9920 return ident; 9921 } 9922 9923 /++ 9924 UDA used to indicate to the [dispatcher] that a trailing slash should always be added to or removed from the url. It will do it as a redirect header as-needed. 9925 +/ 9926 enum AddTrailingSlash; 9927 /// ditto 9928 enum RemoveTrailingSlash; 9929 9930 private auto serveApiInternal(T)(string urlPrefix) { 9931 9932 import arsd.dom; 9933 import arsd.jsvar; 9934 9935 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 9936 string remainingUrl = cgi.pathInfo[urlPrefix.length .. $]; 9937 9938 try { 9939 // see duplicated code below by searching subresource_ctor 9940 // also see mustNotBeSetFromWebParams 9941 9942 static if(is(typeof(T.__ctor) P == __parameters)) { 9943 P params; 9944 9945 foreach(pidx, param; P) { 9946 static if(is(param : Cgi)) { 9947 static assert(!is(param == immutable)); 9948 cast() params[pidx] = cgi; 9949 } else static if(is(param == Session!D, D)) { 9950 static assert(!is(param == immutable)); 9951 cast() params[pidx] = cgi.getSessionObject!D(); 9952 9953 } else { 9954 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 9955 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 9956 static if(is(uda == ifCalledFromWeb!func, alias func)) { 9957 static if(is(typeof(func(cgi)))) 9958 params[pidx] = func(cgi); 9959 else 9960 params[pidx] = func(); 9961 } 9962 } 9963 } else { 9964 9965 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 9966 params[pidx] = param.getAutomaticallyForCgi(cgi); 9967 } else static if(is(param == string)) { 9968 auto ident = nextPieceFromSlash(remainingUrl); 9969 params[pidx] = ident; 9970 } else static assert(0, "illegal type for subresource " ~ param.stringof); 9971 } 9972 } 9973 } 9974 9975 auto obj = new T(params); 9976 } else { 9977 auto obj = new T(); 9978 } 9979 9980 return internalHandlerWithObject(obj, remainingUrl, cgi, presenter); 9981 } catch(Throwable t) { 9982 switch(cgi.request("format", "html")) { 9983 case "html": 9984 static void dummy() {} 9985 presenter.presentExceptionAsHtml!(dummy)(cgi, t, &dummy); 9986 return true; 9987 case "json": 9988 var envelope = var.emptyObject; 9989 envelope.success = false; 9990 envelope.result = null; 9991 envelope.error = t.toString(); 9992 cgi.setResponseContentType("application/json"); 9993 cgi.write(envelope.toJson(), true); 9994 return true; 9995 default: 9996 throw t; 9997 // return true; 9998 } 9999 // return true; 10000 } 10001 10002 assert(0); 10003 } 10004 10005 static bool internalHandlerWithObject(T, Presenter)(T obj, string remainingUrl, Cgi cgi, Presenter presenter) { 10006 10007 obj.initialize(cgi); 10008 10009 /+ 10010 Overload rules: 10011 Any unique combination of HTTP verb and url path can be dispatched to function overloads 10012 statically. 10013 10014 Moreover, some args vs no args can be overloaded dynamically. 10015 +/ 10016 10017 auto methodNameFromUrl = nextPieceFromSlash(remainingUrl); 10018 /+ 10019 auto orig = remainingUrl; 10020 assert(0, 10021 (orig is null ? "__null" : orig) 10022 ~ " .. " ~ 10023 (methodNameFromUrl is null ? "__null" : methodNameFromUrl)); 10024 +/ 10025 10026 if(methodNameFromUrl is null) 10027 methodNameFromUrl = "__null"; 10028 10029 string hack = to!string(cgi.requestMethod) ~ " " ~ methodNameFromUrl; 10030 10031 if(remainingUrl.length) 10032 hack ~= "/"; 10033 10034 switch(hack) { 10035 foreach(methodName; __traits(derivedMembers, T)) 10036 static if(methodName != "__ctor") 10037 foreach(idx, overload; __traits(getOverloads, T, methodName)) { 10038 static if(is(typeof(overload) P == __parameters)) 10039 static if(is(typeof(overload) R == return)) 10040 static if(__traits(getProtection, overload) == "public" || __traits(getProtection, overload) == "export") 10041 { 10042 static foreach(urlNameForMethod; urlNamesForMethod!(overload, urlify(methodName))) 10043 case urlNameForMethod: 10044 10045 static if(is(R : WebObject)) { 10046 // if it returns a WebObject, it is considered a subresource. That means the url is dispatched like the ctor above. 10047 10048 // the only argument it is allowed to take, outside of cgi, session, and set up thingies, is a single string 10049 10050 // subresource_ctor 10051 // also see mustNotBeSetFromWebParams 10052 10053 P params; 10054 10055 string ident; 10056 10057 foreach(pidx, param; P) { 10058 static if(is(param : Cgi)) { 10059 static assert(!is(param == immutable)); 10060 cast() params[pidx] = cgi; 10061 } else static if(is(param == typeof(presenter))) { 10062 cast() param[pidx] = presenter; 10063 } else static if(is(param == Session!D, D)) { 10064 static assert(!is(param == immutable)); 10065 cast() params[pidx] = cgi.getSessionObject!D(); 10066 } else { 10067 static if(hasIfCalledFromWeb!(__traits(getAttributes, P[pidx .. pidx + 1]))) { 10068 foreach(uda; __traits(getAttributes, P[pidx .. pidx + 1])) { 10069 static if(is(uda == ifCalledFromWeb!func, alias func)) { 10070 static if(is(typeof(func(cgi)))) 10071 params[pidx] = func(cgi); 10072 else 10073 params[pidx] = func(); 10074 } 10075 } 10076 } else { 10077 10078 static if(__traits(compiles, { params[pidx] = param.getAutomaticallyForCgi(cgi); } )) { 10079 params[pidx] = param.getAutomaticallyForCgi(cgi); 10080 } else static if(is(param == string)) { 10081 ident = nextPieceFromSlash(remainingUrl); 10082 if(ident is null) { 10083 // trailing slash mandated on subresources 10084 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10085 return true; 10086 } else { 10087 params[pidx] = ident; 10088 } 10089 } else static assert(0, "illegal type for subresource " ~ param.stringof); 10090 } 10091 } 10092 } 10093 10094 auto nobj = (__traits(getOverloads, obj, methodName)[idx])(ident); 10095 return internalHandlerWithObject!(typeof(nobj), Presenter)(nobj, remainingUrl, cgi, presenter); 10096 } else { 10097 // 404 it if any url left - not a subresource means we don't get to play with that! 10098 if(remainingUrl.length) 10099 return false; 10100 10101 bool automaticForm; 10102 10103 foreach(attr; __traits(getAttributes, overload)) 10104 static if(is(attr == AddTrailingSlash)) { 10105 if(remainingUrl is null) { 10106 cgi.setResponseLocation(cgi.pathInfo ~ "/"); 10107 return true; 10108 } 10109 } else static if(is(attr == RemoveTrailingSlash)) { 10110 if(remainingUrl !is null) { 10111 cgi.setResponseLocation(cgi.pathInfo[0 .. lastIndexOf(cgi.pathInfo, "/")]); 10112 return true; 10113 } 10114 10115 } else static if(__traits(isSame, AutomaticForm, attr)) { 10116 automaticForm = true; 10117 } 10118 10119 /+ 10120 int zeroArgOverload = -1; 10121 int overloadCount = cast(int) __traits(getOverloads, T, methodName).length; 10122 bool calledWithZeroArgs = true; 10123 foreach(k, v; cgi.get) 10124 if(k != "format") { 10125 calledWithZeroArgs = false; 10126 break; 10127 } 10128 foreach(k, v; cgi.post) 10129 if(k != "format") { 10130 calledWithZeroArgs = false; 10131 break; 10132 } 10133 10134 // first, we need to go through and see if there is an empty one, since that 10135 // changes inside. But otherwise, all the stuff I care about can be done via 10136 // simple looping (other improper overloads might be flagged for runtime semantic check) 10137 // 10138 // an argument of type Cgi is ignored for these purposes 10139 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10140 static if(is(typeof(overload) P == __parameters)) 10141 static if(P.length == 0) 10142 zeroArgOverload = cast(int) idx; 10143 else static if(P.length == 1 && is(P[0] : Cgi)) 10144 zeroArgOverload = cast(int) idx; 10145 }} 10146 // FIXME: static assert if there are multiple non-zero-arg overloads usable with a single http method. 10147 bool overloadHasBeenCalled = false; 10148 static foreach(idx, overload; __traits(getOverloads, T, methodName)) {{ 10149 bool callFunction = true; 10150 // there is a zero arg overload and this is NOT it, and we have zero args - don't call this 10151 if(overloadCount > 1 && zeroArgOverload != -1 && idx != zeroArgOverload && calledWithZeroArgs) 10152 callFunction = false; 10153 // if this is the zero-arg overload, obviously it cannot be called if we got any args. 10154 if(overloadCount > 1 && idx == zeroArgOverload && !calledWithZeroArgs) 10155 callFunction = false; 10156 10157 // FIXME: so if you just add ?foo it will give the error below even when. this might not be a great idea. 10158 10159 bool hadAnyMethodRestrictions = false; 10160 bool foundAcceptableMethod = false; 10161 foreach(attr; __traits(getAttributes, overload)) { 10162 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10163 hadAnyMethodRestrictions = true; 10164 if(attr == cgi.requestMethod) 10165 foundAcceptableMethod = true; 10166 } 10167 } 10168 10169 if(hadAnyMethodRestrictions && !foundAcceptableMethod) 10170 callFunction = false; 10171 10172 /+ 10173 The overloads we really want to allow are the sane ones 10174 from the web perspective. Which is likely on HTTP verbs, 10175 for the most part, but might also be potentially based on 10176 some args vs zero args, or on argument names. Can't really 10177 do argument types very reliable through the web though; those 10178 should probably be different URLs. 10179 10180 Even names I feel is better done inside the function, so I'm not 10181 going to support that here. But the HTTP verbs and zero vs some 10182 args makes sense - it lets you define custom forms pretty easily. 10183 10184 Moreover, I'm of the opinion that empty overload really only makes 10185 sense on GET for this case. On a POST, it is just a missing argument 10186 exception and that should be handled by the presenter. But meh, I'll 10187 let the user define that, D only allows one empty arg thing anyway 10188 so the method UDAs are irrelevant. 10189 +/ 10190 if(callFunction) 10191 +/ 10192 10193 if(automaticForm && cgi.requestMethod == Cgi.RequestMethod.GET) { 10194 // Should I still show the form on a json thing? idk... 10195 auto ret = presenter.createAutomaticFormForFunction!((__traits(getOverloads, obj, methodName)[idx]))(&(__traits(getOverloads, obj, methodName)[idx])); 10196 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), "html"); 10197 return true; 10198 } 10199 switch(cgi.request("format", defaultFormat!overload())) { 10200 case "html": 10201 // a void return (or typeof(null) lol) means you, the user, is doing it yourself. Gives full control. 10202 try { 10203 10204 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10205 presenter.presentSuccessfulReturn(cgi, ret, presenter.methodMeta!(__traits(getOverloads, obj, methodName)[idx]), "html"); 10206 } catch(Throwable t) { 10207 presenter.presentExceptionAsHtml!(__traits(getOverloads, obj, methodName)[idx])(cgi, t, &(__traits(getOverloads, obj, methodName)[idx])); 10208 } 10209 return true; 10210 case "json": 10211 auto ret = callFromCgi!(__traits(getOverloads, obj, methodName)[idx])(&(__traits(getOverloads, obj, methodName)[idx]), cgi); 10212 static if(is(typeof(ret) == MultipleResponses!Types, Types...)) { 10213 var json; 10214 foreach(index, type; Types) { 10215 if(ret.contains == index) 10216 json = ret.payload[index]; 10217 } 10218 } else { 10219 var json = ret; 10220 } 10221 var envelope = json; // var.emptyObject; 10222 /* 10223 envelope.success = true; 10224 envelope.result = json; 10225 envelope.error = null; 10226 */ 10227 cgi.setResponseContentType("application/json"); 10228 cgi.write(envelope.toJson(), true); 10229 return true; 10230 default: 10231 cgi.setResponseStatus("406 Not Acceptable"); // not exactly but sort of. 10232 return true; 10233 } 10234 //}} 10235 10236 //cgi.header("Accept: POST"); // FIXME list the real thing 10237 //cgi.setResponseStatus("405 Method Not Allowed"); // again, not exactly, but sort of. no overload matched our args, almost certainly due to http verb filtering. 10238 //return true; 10239 } 10240 } 10241 } 10242 case "GET script.js": 10243 cgi.setResponseContentType("text/javascript"); 10244 cgi.gzipResponse = true; 10245 cgi.write(presenter.script(), true); 10246 return true; 10247 case "GET style.css": 10248 cgi.setResponseContentType("text/css"); 10249 cgi.gzipResponse = true; 10250 cgi.write(presenter.style(), true); 10251 return true; 10252 default: 10253 return false; 10254 } 10255 10256 assert(0); 10257 } 10258 return DispatcherDefinition!internalHandler(urlPrefix, false); 10259 } 10260 10261 string defaultFormat(alias method)() { 10262 bool nonConstConditionForWorkingAroundASpuriousDmdWarning = true; 10263 foreach(attr; __traits(getAttributes, method)) { 10264 static if(is(typeof(attr) == DefaultFormat)) { 10265 if(nonConstConditionForWorkingAroundASpuriousDmdWarning) 10266 return attr.value; 10267 } 10268 } 10269 return "html"; 10270 } 10271 10272 struct Paginated(T) { 10273 T[] items; 10274 string nextPageUrl; 10275 } 10276 10277 template urlNamesForMethod(alias method, string default_) { 10278 string[] helper() { 10279 auto verb = Cgi.RequestMethod.GET; 10280 bool foundVerb = false; 10281 bool foundNoun = false; 10282 10283 string def = default_; 10284 10285 bool hasAutomaticForm = false; 10286 10287 foreach(attr; __traits(getAttributes, method)) { 10288 static if(is(typeof(attr) == Cgi.RequestMethod)) { 10289 verb = attr; 10290 if(foundVerb) 10291 assert(0, "Multiple http verbs on one function is not currently supported"); 10292 foundVerb = true; 10293 } 10294 static if(is(typeof(attr) == UrlName)) { 10295 if(foundNoun) 10296 assert(0, "Multiple url names on one function is not currently supported"); 10297 foundNoun = true; 10298 def = attr.name; 10299 } 10300 static if(__traits(isSame, attr, AutomaticForm)) { 10301 hasAutomaticForm = true; 10302 } 10303 } 10304 10305 if(def is null) 10306 def = "__null"; 10307 10308 string[] ret; 10309 10310 static if(is(typeof(method) R == return)) { 10311 static if(is(R : WebObject)) { 10312 def ~= "/"; 10313 foreach(v; __traits(allMembers, Cgi.RequestMethod)) 10314 ret ~= v ~ " " ~ def; 10315 } else { 10316 if(hasAutomaticForm) { 10317 ret ~= "GET " ~ def; 10318 ret ~= "POST " ~ def; 10319 } else { 10320 ret ~= to!string(verb) ~ " " ~ def; 10321 } 10322 } 10323 } else static assert(0); 10324 10325 return ret; 10326 } 10327 enum urlNamesForMethod = helper(); 10328 } 10329 10330 10331 enum AccessCheck { 10332 allowed, 10333 denied, 10334 nonExistant, 10335 } 10336 10337 enum Operation { 10338 show, 10339 create, 10340 replace, 10341 remove, 10342 update 10343 } 10344 10345 enum UpdateResult { 10346 accessDenied, 10347 noSuchResource, 10348 success, 10349 failure, 10350 unnecessary 10351 } 10352 10353 enum ValidationResult { 10354 valid, 10355 invalid 10356 } 10357 10358 10359 /++ 10360 The base of all REST objects, to be used with [serveRestObject] and [serveRestCollectionOf]. 10361 10362 WARNING: this is not stable. 10363 +/ 10364 class RestObject(CRTP) : WebObject { 10365 10366 import arsd.dom; 10367 import arsd.jsvar; 10368 10369 /// Prepare the object to be shown. 10370 void show() {} 10371 /// ditto 10372 void show(string urlId) { 10373 load(urlId); 10374 show(); 10375 } 10376 10377 /// Override this to provide access control to this object. 10378 AccessCheck accessCheck(string urlId, Operation operation) { 10379 return AccessCheck.allowed; 10380 } 10381 10382 ValidationResult validate() { 10383 // FIXME 10384 return ValidationResult.valid; 10385 } 10386 10387 string getUrlSlug() { 10388 import std.conv; 10389 static if(is(typeof(CRTP.id))) 10390 return to!string((cast(CRTP) this).id); 10391 else 10392 return null; 10393 } 10394 10395 // The functions with more arguments are the low-level ones, 10396 // they forward to the ones with fewer arguments by default. 10397 10398 // POST on a parent collection - this is called from a collection class after the members are updated 10399 /++ 10400 Given a populated object, this creates a new entry. Returns the url identifier 10401 of the new object. 10402 +/ 10403 string create(scope void delegate() applyChanges) { 10404 applyChanges(); 10405 save(); 10406 return getUrlSlug(); 10407 } 10408 10409 void replace() { 10410 save(); 10411 } 10412 void replace(string urlId, scope void delegate() applyChanges) { 10413 load(urlId); 10414 applyChanges(); 10415 replace(); 10416 } 10417 10418 void update(string[] fieldList) { 10419 save(); 10420 } 10421 void update(string urlId, scope void delegate() applyChanges, string[] fieldList) { 10422 load(urlId); 10423 applyChanges(); 10424 update(fieldList); 10425 } 10426 10427 void remove() {} 10428 10429 void remove(string urlId) { 10430 load(urlId); 10431 remove(); 10432 } 10433 10434 abstract void load(string urlId); 10435 abstract void save(); 10436 10437 Element toHtml(Presenter)(Presenter presenter) { 10438 import arsd.dom; 10439 import std.conv; 10440 auto obj = cast(CRTP) this; 10441 auto div = Element.make("div"); 10442 div.addClass("Dclass_" ~ CRTP.stringof); 10443 div.dataset.url = getUrlSlug(); 10444 bool first = true; 10445 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 10446 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10447 if(!first) div.addChild("br"); else first = false; 10448 div.appendChild(presenter.formatReturnValueAsHtml(__traits(getMember, obj, memberName))); 10449 } 10450 return div; 10451 } 10452 10453 var toJson() { 10454 import arsd.jsvar; 10455 var v = var.emptyObject(); 10456 auto obj = cast(CRTP) this; 10457 foreach(idx, memberName; __traits(derivedMembers, CRTP)) 10458 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10459 v[memberName] = __traits(getMember, obj, memberName); 10460 } 10461 return v; 10462 } 10463 10464 /+ 10465 auto structOf(this This) { 10466 10467 } 10468 +/ 10469 } 10470 10471 // FIXME XSRF token, prolly can just put in a cookie and then it needs to be copied to header or form hidden value 10472 // https://use-the-index-luke.com/sql/partial-results/fetch-next-page 10473 10474 /++ 10475 Base class for REST collections. 10476 +/ 10477 class CollectionOf(Obj) : RestObject!(CollectionOf) { 10478 /// You might subclass this and use the cgi object's query params 10479 /// to implement a search filter, for example. 10480 /// 10481 /// FIXME: design a way to auto-generate that form 10482 /// (other than using the WebObject thing above lol 10483 // it'll prolly just be some searchParams UDA or maybe an enum. 10484 // 10485 // pagination too perhaps. 10486 // 10487 // and sorting too 10488 IndexResult index() { return IndexResult.init; } 10489 10490 string[] sortableFields() { return null; } 10491 string[] searchableFields() { return null; } 10492 10493 struct IndexResult { 10494 Obj[] results; 10495 10496 string[] sortableFields; 10497 10498 string previousPageIdentifier; 10499 string nextPageIdentifier; 10500 string firstPageIdentifier; 10501 string lastPageIdentifier; 10502 10503 int numberOfPages; 10504 } 10505 10506 override string create(scope void delegate() applyChanges) { assert(0); } 10507 override void load(string urlId) { assert(0); } 10508 override void save() { assert(0); } 10509 override void show() { 10510 index(); 10511 } 10512 override void show(string urlId) { 10513 show(); 10514 } 10515 10516 /// Proxy POST requests (create calls) to the child collection 10517 alias PostProxy = Obj; 10518 } 10519 10520 /++ 10521 Serves a REST object, similar to a Ruby on Rails resource. 10522 10523 You put data members in your class. cgi.d will automatically make something out of those. 10524 10525 It will call your constructor with the ID from the URL. This may be null. 10526 It will then populate the data members from the request. 10527 It will then call a method, if present, telling what happened. You don't need to write these! 10528 It finally returns a reply. 10529 10530 Your methods are passed a list of fields it actually set. 10531 10532 The URL mapping - despite my general skepticism of the wisdom - matches up with what most REST 10533 APIs I have used seem to follow. (I REALLY want to put trailing slashes on it though. Works better 10534 with relative linking. But meh.) 10535 10536 GET /items -> index. all values not set. 10537 GET /items/id -> get. only ID will be set, other params ignored. 10538 POST /items -> create. values set as given 10539 PUT /items/id -> replace. values set as given 10540 or POST /items/id with cgi.post["_method"] (thus urlencoded or multipart content-type) set to "PUT" to work around browser/html limitation 10541 a GET with cgi.get["_method"] (in the url) set to "PUT" will render a form. 10542 PATCH /items/id -> update. values set as given, list of changed fields passed 10543 or POST /items/id with cgi.post["_method"] == "PATCH" 10544 DELETE /items/id -> destroy. only ID guaranteed to be set 10545 or POST /items/id with cgi.post["_method"] == "DELETE" 10546 10547 Following the stupid convention, there will never be a trailing slash here, and if it is there, it will 10548 redirect you away from it. 10549 10550 API clients should set the `Accept` HTTP header to application/json or the cgi.get["_format"] = "json" var. 10551 10552 I will also let you change the default, if you must. 10553 10554 // One add-on is validation. You can issue a HTTP GET to a resource with _method = VALIDATE to check potential changes. 10555 10556 You can define sub-resources on your object inside the object. These sub-resources are also REST objects 10557 that follow the same thing. They may be individual resources or collections themselves. 10558 10559 Your class is expected to have at least the following methods: 10560 10561 FIXME: i kinda wanna add a routes object to the initialize call 10562 10563 create 10564 Create returns the new address on success, some code on failure. 10565 show 10566 index 10567 update 10568 remove 10569 10570 You will want to be able to customize the HTTP, HTML, and JSON returns but generally shouldn't have to - the defaults 10571 should usually work. The returned JSON will include a field "href" on all returned objects along with "id". Or omething like that. 10572 10573 Usage of this function will add a dependency on [arsd.dom] and [arsd.jsvar]. 10574 10575 NOT IMPLEMENTED 10576 10577 10578 Really, a collection is a resource with a bunch of subresources. 10579 10580 GET /items 10581 index because it is GET on the top resource 10582 10583 GET /items/foo 10584 item but different than items? 10585 10586 class Items { 10587 10588 } 10589 10590 ... but meh, a collection can be automated. not worth making it 10591 a separate thing, let's look at a real example. Users has many 10592 items and a virtual one, /users/current. 10593 10594 the individual users have properties and two sub-resources: 10595 session, which is just one, and comments, a collection. 10596 10597 class User : RestObject!() { // no parent 10598 int id; 10599 string name; 10600 10601 // the default implementations of the urlId ones is to call load(that_id) then call the arg-less one. 10602 // but you can override them to do it differently. 10603 10604 // any member which is of type RestObject can be linked automatically via href btw. 10605 10606 void show() {} 10607 void show(string urlId) {} // automated! GET of this specific thing 10608 void create() {} // POST on a parent collection - this is called from a collection class after the members are updated 10609 void replace(string urlId) {} // this is the PUT; really, it just updates all fields. 10610 void update(string urlId, string[] fieldList) {} // PATCH, it updates some fields. 10611 void remove(string urlId) {} // DELETE 10612 10613 void load(string urlId) {} // the default implementation of show() populates the id, then 10614 10615 this() {} 10616 10617 mixin Subresource!Session; 10618 mixin Subresource!Comment; 10619 } 10620 10621 class Session : RestObject!() { 10622 // the parent object may not be fully constructed/loaded 10623 this(User parent) {} 10624 10625 } 10626 10627 class Comment : CollectionOf!Comment { 10628 this(User parent) {} 10629 } 10630 10631 class Users : CollectionOf!User { 10632 // but you don't strictly need ANYTHING on a collection; it will just... collect. Implement the subobjects. 10633 void index() {} // GET on this specific thing; just like show really, just different name for the different semantics. 10634 User create() {} // You MAY implement this, but the default is to create a new object, populate it from args, and then call create() on the child 10635 } 10636 10637 +/ 10638 auto serveRestObject(T)(string urlPrefix) { 10639 assert(urlPrefix[0] == '/'); 10640 assert(urlPrefix[$ - 1] != '/', "Do NOT use a trailing slash on REST objects."); 10641 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, immutable void* details) { 10642 string url = cgi.pathInfo[urlPrefix.length .. $]; 10643 10644 if(url.length && url[$ - 1] == '/') { 10645 // remove the final slash... 10646 cgi.setResponseLocation(cgi.scriptName ~ cgi.pathInfo[0 .. $ - 1]); 10647 return true; 10648 } 10649 10650 return restObjectServeHandler!T(cgi, presenter, url); 10651 } 10652 return DispatcherDefinition!internalHandler(urlPrefix, false); 10653 } 10654 10655 /+ 10656 /// Convenience method for serving a collection. It will be named the same 10657 /// as type T, just with an s at the end. If you need any further, just 10658 /// write the class yourself. 10659 auto serveRestCollectionOf(T)(string urlPrefix) { 10660 assert(urlPrefix[0] == '/'); 10661 mixin(`static class `~T.stringof~`s : CollectionOf!(T) {}`); 10662 return serveRestObject!(mixin(T.stringof ~ "s"))(urlPrefix); 10663 } 10664 +/ 10665 10666 bool restObjectServeHandler(T, Presenter)(Cgi cgi, Presenter presenter, string url) { 10667 string urlId = null; 10668 if(url.length && url[0] == '/') { 10669 // asking for a subobject 10670 urlId = url[1 .. $]; 10671 foreach(idx, ch; urlId) { 10672 if(ch == '/') { 10673 urlId = urlId[0 .. idx]; 10674 break; 10675 } 10676 } 10677 } 10678 10679 // FIXME handle other subresources 10680 10681 static if(is(T : CollectionOf!(C), C)) { 10682 if(urlId !is null) { 10683 return restObjectServeHandler!(C, Presenter)(cgi, presenter, url); // FIXME? urlId); 10684 } 10685 } 10686 10687 // FIXME: support precondition failed, if-modified-since, expectation failed, etc. 10688 10689 auto obj = new T(); 10690 obj.initialize(cgi); 10691 // FIXME: populate reflection info delegates 10692 10693 10694 // FIXME: I am not happy with this. 10695 switch(urlId) { 10696 case "script.js": 10697 cgi.setResponseContentType("text/javascript"); 10698 cgi.gzipResponse = true; 10699 cgi.write(presenter.script(), true); 10700 return true; 10701 case "style.css": 10702 cgi.setResponseContentType("text/css"); 10703 cgi.gzipResponse = true; 10704 cgi.write(presenter.style(), true); 10705 return true; 10706 default: 10707 // intentionally blank 10708 } 10709 10710 10711 10712 10713 static void applyChangesTemplate(Obj)(Cgi cgi, Obj obj) { 10714 foreach(idx, memberName; __traits(derivedMembers, Obj)) 10715 static if(__traits(compiles, __traits(getMember, obj, memberName).offsetof)) { 10716 __traits(getMember, obj, memberName) = cgi.request(memberName, __traits(getMember, obj, memberName)); 10717 } 10718 } 10719 void applyChanges() { 10720 applyChangesTemplate(cgi, obj); 10721 } 10722 10723 string[] modifiedList; 10724 10725 void writeObject(bool addFormLinks) { 10726 if(cgi.request("format") == "json") { 10727 cgi.setResponseContentType("application/json"); 10728 cgi.write(obj.toJson().toString, true); 10729 } else { 10730 auto container = presenter.htmlContainer(); 10731 if(addFormLinks) { 10732 static if(is(T : CollectionOf!(C), C)) 10733 container.appendHtml(` 10734 <form> 10735 <button type="submit" name="_method" value="POST">Create New</button> 10736 </form> 10737 `); 10738 else 10739 container.appendHtml(` 10740 <a href="..">Back</a> 10741 <form> 10742 <button type="submit" name="_method" value="PATCH">Edit</button> 10743 <button type="submit" name="_method" value="DELETE">Delete</button> 10744 </form> 10745 `); 10746 } 10747 container.appendChild(obj.toHtml(presenter)); 10748 cgi.write(container.parentDocument.toString, true); 10749 } 10750 } 10751 10752 // FIXME: I think I need a set type in here.... 10753 // it will be nice to pass sets of members. 10754 10755 try 10756 switch(cgi.requestMethod) { 10757 case Cgi.RequestMethod.GET: 10758 // I could prolly use template this parameters in the implementation above for some reflection stuff. 10759 // sure, it doesn't automatically work in subclasses... but I instantiate here anyway... 10760 10761 // automatic forms here for usable basic auto site from browser. 10762 // even if the format is json, it could actually send out the links and formats, but really there i'ma be meh. 10763 switch(cgi.request("_method", "GET")) { 10764 case "GET": 10765 static if(is(T : CollectionOf!(C), C)) { 10766 auto results = obj.index(); 10767 if(cgi.request("format", "html") == "html") { 10768 auto container = presenter.htmlContainer(); 10769 auto html = presenter.formatReturnValueAsHtml(results.results); 10770 container.appendHtml(` 10771 <form> 10772 <button type="submit" name="_method" value="POST">Create New</button> 10773 </form> 10774 `); 10775 10776 container.appendChild(html); 10777 cgi.write(container.parentDocument.toString, true); 10778 } else { 10779 cgi.setResponseContentType("application/json"); 10780 import arsd.jsvar; 10781 var json = var.emptyArray; 10782 foreach(r; results.results) { 10783 var o = var.emptyObject; 10784 foreach(idx, memberName; __traits(derivedMembers, typeof(r))) 10785 static if(__traits(compiles, __traits(getMember, r, memberName).offsetof)) { 10786 o[memberName] = __traits(getMember, r, memberName); 10787 } 10788 10789 json ~= o; 10790 } 10791 cgi.write(json.toJson(), true); 10792 } 10793 } else { 10794 obj.show(urlId); 10795 writeObject(true); 10796 } 10797 break; 10798 case "PATCH": 10799 obj.load(urlId); 10800 goto case; 10801 case "PUT": 10802 case "POST": 10803 // an editing form for the object 10804 auto container = presenter.htmlContainer(); 10805 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 10806 auto form = (cgi.request("_method") == "POST") ? presenter.createAutomaticFormForObject(new obj.PostProxy()) : presenter.createAutomaticFormForObject(obj); 10807 } else { 10808 auto form = presenter.createAutomaticFormForObject(obj); 10809 } 10810 form.attrs.method = "POST"; 10811 form.setValue("_method", cgi.request("_method", "GET")); 10812 container.appendChild(form); 10813 cgi.write(container.parentDocument.toString(), true); 10814 break; 10815 case "DELETE": 10816 // FIXME: a delete form for the object (can be phrased "are you sure?") 10817 auto container = presenter.htmlContainer(); 10818 container.appendHtml(` 10819 <form method="POST"> 10820 Are you sure you want to delete this item? 10821 <input type="hidden" name="_method" value="DELETE" /> 10822 <input type="submit" value="Yes, Delete It" /> 10823 </form> 10824 10825 `); 10826 cgi.write(container.parentDocument.toString(), true); 10827 break; 10828 default: 10829 cgi.write("bad method\n", true); 10830 } 10831 break; 10832 case Cgi.RequestMethod.POST: 10833 // this is to allow compatibility with HTML forms 10834 switch(cgi.request("_method", "POST")) { 10835 case "PUT": 10836 goto PUT; 10837 case "PATCH": 10838 goto PATCH; 10839 case "DELETE": 10840 goto DELETE; 10841 case "POST": 10842 static if(__traits(compiles, () { auto o = new obj.PostProxy(); })) { 10843 auto p = new obj.PostProxy(); 10844 void specialApplyChanges() { 10845 applyChangesTemplate(cgi, p); 10846 } 10847 string n = p.create(&specialApplyChanges); 10848 } else { 10849 string n = obj.create(&applyChanges); 10850 } 10851 10852 auto newUrl = cgi.scriptName ~ cgi.pathInfo ~ "/" ~ n; 10853 cgi.setResponseLocation(newUrl); 10854 cgi.setResponseStatus("201 Created"); 10855 cgi.write(`The object has been created.`); 10856 break; 10857 default: 10858 cgi.write("bad method\n", true); 10859 } 10860 // FIXME this should be valid on the collection, but not the child.... 10861 // 303 See Other 10862 break; 10863 case Cgi.RequestMethod.PUT: 10864 PUT: 10865 obj.replace(urlId, &applyChanges); 10866 writeObject(false); 10867 break; 10868 case Cgi.RequestMethod.PATCH: 10869 PATCH: 10870 obj.update(urlId, &applyChanges, modifiedList); 10871 writeObject(false); 10872 break; 10873 case Cgi.RequestMethod.DELETE: 10874 DELETE: 10875 obj.remove(urlId); 10876 cgi.setResponseStatus("204 No Content"); 10877 break; 10878 default: 10879 // FIXME: OPTIONS, HEAD 10880 } 10881 catch(Throwable t) { 10882 presenter.presentExceptionAsHtml!(DUMMY)(cgi, t, null); 10883 } 10884 10885 return true; 10886 } 10887 10888 struct DUMMY {} 10889 10890 /+ 10891 struct SetOfFields(T) { 10892 private void[0][string] storage; 10893 void set(string what) { 10894 //storage[what] = 10895 } 10896 void unset(string what) {} 10897 void setAll() {} 10898 void unsetAll() {} 10899 bool isPresent(string what) { return false; } 10900 } 10901 +/ 10902 10903 /+ 10904 enum readonly; 10905 enum hideonindex; 10906 +/ 10907 10908 /++ 10909 Serves a static file. To be used with [dispatcher]. 10910 10911 See_Also: [serveApi], [serveRestObject], [dispatcher], [serveRedirect] 10912 +/ 10913 auto serveStaticFile(string urlPrefix, string filename = null, string contentType = null) { 10914 // https://baus.net/on-tcp_cork/ 10915 // man 2 sendfile 10916 assert(urlPrefix[0] == '/'); 10917 if(filename is null) 10918 filename = decodeComponent(urlPrefix[1 .. $]); // FIXME is this actually correct? 10919 if(contentType is null) { 10920 contentType = contentTypeFromFileExtension(filename); 10921 } 10922 10923 static struct DispatcherDetails { 10924 string filename; 10925 string contentType; 10926 } 10927 10928 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 10929 if(details.contentType.indexOf("image/") == 0) 10930 cgi.setCache(true); 10931 cgi.setResponseContentType(details.contentType); 10932 cgi.write(std.file.read(details.filename), true); 10933 return true; 10934 } 10935 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(filename, contentType)); 10936 } 10937 10938 /++ 10939 Serves static data. To be used with [dispatcher]. 10940 10941 History: 10942 Added October 31, 2021 10943 +/ 10944 auto serveStaticData(string urlPrefix, immutable(void)[] data, string contentType = null) { 10945 assert(urlPrefix[0] == '/'); 10946 if(contentType is null) { 10947 contentType = contentTypeFromFileExtension(urlPrefix); 10948 } 10949 10950 static struct DispatcherDetails { 10951 immutable(void)[] data; 10952 string contentType; 10953 } 10954 10955 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 10956 cgi.setCache(true); 10957 cgi.setResponseContentType(details.contentType); 10958 cgi.write(details.data, true); 10959 return true; 10960 } 10961 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(data, contentType)); 10962 } 10963 10964 string contentTypeFromFileExtension(string filename) { 10965 if(filename.endsWith(".png")) 10966 return "image/png"; 10967 if(filename.endsWith(".apng")) 10968 return "image/apng"; 10969 if(filename.endsWith(".svg")) 10970 return "image/svg+xml"; 10971 if(filename.endsWith(".jpg")) 10972 return "image/jpeg"; 10973 if(filename.endsWith(".html")) 10974 return "text/html"; 10975 if(filename.endsWith(".css")) 10976 return "text/css"; 10977 if(filename.endsWith(".js")) 10978 return "application/javascript"; 10979 if(filename.endsWith(".wasm")) 10980 return "application/wasm"; 10981 if(filename.endsWith(".mp3")) 10982 return "audio/mpeg"; 10983 return null; 10984 } 10985 10986 /// This serves a directory full of static files, figuring out the content-types from file extensions. 10987 /// It does not let you to descend into subdirectories (or ascend out of it, of course) 10988 auto serveStaticFileDirectory(string urlPrefix, string directory = null) { 10989 assert(urlPrefix[0] == '/'); 10990 assert(urlPrefix[$-1] == '/'); 10991 10992 static struct DispatcherDetails { 10993 string directory; 10994 } 10995 10996 if(directory is null) 10997 directory = urlPrefix[1 .. $]; 10998 10999 assert(directory[$-1] == '/'); 11000 11001 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11002 auto file = decodeComponent(cgi.pathInfo[urlPrefix.length .. $]); // FIXME: is this actually correct 11003 if(file.indexOf("/") != -1 || file.indexOf("\\") != -1) 11004 return false; 11005 11006 auto contentType = contentTypeFromFileExtension(file); 11007 11008 auto fn = details.directory ~ file; 11009 if(std.file.exists(fn)) { 11010 //if(contentType.indexOf("image/") == 0) 11011 //cgi.setCache(true); 11012 //else if(contentType.indexOf("audio/") == 0) 11013 cgi.setCache(true); 11014 cgi.setResponseContentType(contentType); 11015 cgi.write(std.file.read(fn), true); 11016 return true; 11017 } else { 11018 return false; 11019 } 11020 } 11021 11022 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, false, DispatcherDetails(directory)); 11023 } 11024 11025 /++ 11026 Redirects one url to another 11027 11028 See_Also: [dispatcher], [serveStaticFile] 11029 +/ 11030 auto serveRedirect(string urlPrefix, string redirectTo, int code = 303) { 11031 assert(urlPrefix[0] == '/'); 11032 static struct DispatcherDetails { 11033 string redirectTo; 11034 string code; 11035 } 11036 11037 static bool internalHandler(string urlPrefix, Cgi cgi, Object presenter, DispatcherDetails details) { 11038 cgi.setResponseLocation(details.redirectTo, true, details.code); 11039 return true; 11040 } 11041 11042 11043 return DispatcherDefinition!(internalHandler, DispatcherDetails)(urlPrefix, true, DispatcherDetails(redirectTo, getHttpCodeText(code))); 11044 } 11045 11046 /// Used exclusively with `dispatchTo` 11047 struct DispatcherData(Presenter) { 11048 Cgi cgi; /// You can use this cgi object. 11049 Presenter presenter; /// This is the presenter from top level, and will be forwarded to the sub-dispatcher. 11050 size_t pathInfoStart; /// This is forwarded to the sub-dispatcher. It may be marked private later, or at least read-only. 11051 } 11052 11053 /++ 11054 Dispatches the URL to a specific function. 11055 +/ 11056 auto handleWith(alias handler)(string urlPrefix) { 11057 // cuz I'm too lazy to do it better right now 11058 static class Hack : WebObject { 11059 static import std.traits; 11060 @UrlName("") 11061 auto handle(std.traits.Parameters!handler args) { 11062 return handler(args); 11063 } 11064 } 11065 11066 return urlPrefix.serveApiInternal!Hack; 11067 } 11068 11069 /++ 11070 Dispatches the URL (and anything under it) to another dispatcher function. The function should look something like this: 11071 11072 --- 11073 bool other(DD)(DD dd) { 11074 return dd.dispatcher!( 11075 "/whatever".serveRedirect("/success"), 11076 "/api/".serveApi!MyClass 11077 ); 11078 } 11079 --- 11080 11081 The `DD` in there will be an instance of [DispatcherData] which you can inspect, or forward to another dispatcher 11082 here. It is a template to account for any Presenter type, so you can do compile-time analysis in your presenters. 11083 Or, of course, you could just use the exact type in your own code. 11084 11085 You return true if you handle the given url, or false if not. Just returning the result of [dispatcher] will do a 11086 good job. 11087 11088 11089 +/ 11090 auto dispatchTo(alias handler)(string urlPrefix) { 11091 assert(urlPrefix[0] == '/'); 11092 assert(urlPrefix[$-1] != '/'); 11093 static bool internalHandler(Presenter)(string urlPrefix, Cgi cgi, Presenter presenter, const void* details) { 11094 return handler(DispatcherData!Presenter(cgi, presenter, urlPrefix.length)); 11095 } 11096 11097 return DispatcherDefinition!(internalHandler)(urlPrefix, false); 11098 } 11099 11100 /+ 11101 /++ 11102 See [serveStaticFile] if you want to serve a file off disk. 11103 +/ 11104 auto serveStaticData(string urlPrefix, const(void)[] data, string contentType) { 11105 11106 } 11107 +/ 11108 11109 /++ 11110 A URL dispatcher. 11111 11112 --- 11113 if(cgi.dispatcher!( 11114 "/api/".serveApi!MyApiClass, 11115 "/objects/lol".serveRestObject!MyRestObject, 11116 "/file.js".serveStaticFile, 11117 "/admin/".dispatchTo!adminHandler 11118 )) return; 11119 --- 11120 11121 11122 You define a series of url prefixes followed by handlers. 11123 11124 [dispatchTo] will send the request to another function for handling. 11125 You may want to do different pre- and post- processing there, for example, 11126 an authorization check and different page layout. You can use different 11127 presenters and different function chains. NOT IMPLEMENTED 11128 +/ 11129 template dispatcher(definitions...) { 11130 bool dispatcher(Presenter)(Cgi cgi, Presenter presenterArg = null) { 11131 static if(is(Presenter == typeof(null))) { 11132 static class GenericWebPresenter : WebPresenter!(GenericWebPresenter) {} 11133 auto presenter = new GenericWebPresenter(); 11134 } else 11135 alias presenter = presenterArg; 11136 11137 return dispatcher(DispatcherData!(typeof(presenter))(cgi, presenter, 0)); 11138 } 11139 11140 bool dispatcher(DispatcherData)(DispatcherData dispatcherData) if(!is(DispatcherData : Cgi)) { 11141 // I can prolly make this more efficient later but meh. 11142 foreach(definition; definitions) { 11143 if(definition.rejectFurther) { 11144 if(dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $] == definition.urlPrefix) { 11145 auto ret = definition.handler( 11146 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11147 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11148 if(ret) 11149 return true; 11150 } 11151 } else if( 11152 dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart .. $].startsWith(definition.urlPrefix) && 11153 // cgi.d dispatcher urls must be complete or have a /; 11154 // "foo" -> thing should NOT match "foobar", just "foo" or "foo/thing" 11155 (definition.urlPrefix[$-1] == '/' || (dispatcherData.pathInfoStart + definition.urlPrefix.length) == dispatcherData.cgi.pathInfo.length 11156 || dispatcherData.cgi.pathInfo[dispatcherData.pathInfoStart + definition.urlPrefix.length] == '/') 11157 ) { 11158 auto ret = definition.handler( 11159 dispatcherData.cgi.pathInfo[0 .. dispatcherData.pathInfoStart + definition.urlPrefix.length], 11160 dispatcherData.cgi, dispatcherData.presenter, definition.details); 11161 if(ret) 11162 return true; 11163 } 11164 } 11165 return false; 11166 } 11167 } 11168 11169 }); 11170 11171 private struct StackBuffer { 11172 char[1024] initial = void; 11173 char[] buffer; 11174 size_t position; 11175 11176 this(int a) { 11177 buffer = initial[]; 11178 position = 0; 11179 } 11180 11181 void add(in char[] what) { 11182 if(position + what.length > buffer.length) 11183 buffer.length = position + what.length + 1024; // reallocate with GC to handle special cases 11184 buffer[position .. position + what.length] = what[]; 11185 position += what.length; 11186 } 11187 11188 void add(in char[] w1, in char[] w2, in char[] w3 = null) { 11189 add(w1); 11190 add(w2); 11191 add(w3); 11192 } 11193 11194 void add(long v) { 11195 char[16] buffer = void; 11196 auto pos = buffer.length; 11197 bool negative; 11198 if(v < 0) { 11199 negative = true; 11200 v = -v; 11201 } 11202 do { 11203 buffer[--pos] = cast(char) (v % 10 + '0'); 11204 v /= 10; 11205 } while(v); 11206 11207 if(negative) 11208 buffer[--pos] = '-'; 11209 11210 auto res = buffer[pos .. $]; 11211 11212 add(res[]); 11213 } 11214 11215 char[] get() @nogc { 11216 return buffer[0 .. position]; 11217 } 11218 } 11219 11220 // duplicated in http2.d 11221 private static string getHttpCodeText(int code) pure nothrow @nogc { 11222 switch(code) { 11223 case 200: return "200 OK"; 11224 case 201: return "201 Created"; 11225 case 202: return "202 Accepted"; 11226 case 203: return "203 Non-Authoritative Information"; 11227 case 204: return "204 No Content"; 11228 case 205: return "205 Reset Content"; 11229 case 206: return "206 Partial Content"; 11230 // 11231 case 300: return "300 Multiple Choices"; 11232 case 301: return "301 Moved Permanently"; 11233 case 302: return "302 Found"; 11234 case 303: return "303 See Other"; 11235 case 304: return "304 Not Modified"; 11236 case 305: return "305 Use Proxy"; 11237 case 307: return "307 Temporary Redirect"; 11238 case 308: return "308 Permanent Redirect"; 11239 11240 // 11241 case 400: return "400 Bad Request"; 11242 case 401: return "401 Unauthorized"; 11243 case 402: return "402 Payment Required"; 11244 case 403: return "403 Forbidden"; 11245 case 404: return "404 Not Found"; 11246 case 405: return "405 Method Not Allowed"; 11247 case 406: return "406 Not Acceptable"; 11248 case 407: return "407 Proxy Authentication Required"; 11249 case 408: return "408 Request Timeout"; 11250 case 409: return "409 Conflict"; 11251 case 410: return "410 Gone"; 11252 case 411: return "411 Length Required"; 11253 case 412: return "412 Precondition Failed"; 11254 case 413: return "413 Payload Too Large"; 11255 case 414: return "414 URI Too Long"; 11256 case 415: return "415 Unsupported Media Type"; 11257 case 416: return "416 Range Not Satisfiable"; 11258 case 417: return "417 Expectation Failed"; 11259 case 418: return "418 I'm a teapot"; 11260 case 421: return "421 Misdirected Request"; 11261 case 422: return "422 Unprocessable Entity (WebDAV)"; 11262 case 423: return "423 Locked (WebDAV)"; 11263 case 424: return "424 Failed Dependency (WebDAV)"; 11264 case 425: return "425 Too Early"; 11265 case 426: return "426 Upgrade Required"; 11266 case 428: return "428 Precondition Required"; 11267 case 431: return "431 Request Header Fields Too Large"; 11268 case 451: return "451 Unavailable For Legal Reasons"; 11269 11270 case 500: return "500 Internal Server Error"; 11271 case 501: return "501 Not Implemented"; 11272 case 502: return "502 Bad Gateway"; 11273 case 503: return "503 Service Unavailable"; 11274 case 504: return "504 Gateway Timeout"; 11275 case 505: return "505 HTTP Version Not Supported"; 11276 case 506: return "506 Variant Also Negotiates"; 11277 case 507: return "507 Insufficient Storage (WebDAV)"; 11278 case 508: return "508 Loop Detected (WebDAV)"; 11279 case 510: return "510 Not Extended"; 11280 case 511: return "511 Network Authentication Required"; 11281 // 11282 default: assert(0, "Unsupported http code"); 11283 } 11284 } 11285 11286 11287 /+ 11288 /++ 11289 This is the beginnings of my web.d 2.0 - it dispatches web requests to a class object. 11290 11291 It relies on jsvar.d and dom.d. 11292 11293 11294 You can get javascript out of it to call. The generated functions need to look 11295 like 11296 11297 function name(a,b,c,d,e) { 11298 return _call("name", {"realName":a,"sds":b}); 11299 } 11300 11301 And _call returns an object you can call or set up or whatever. 11302 +/ 11303 bool apiDispatcher()(Cgi cgi) { 11304 import arsd.jsvar; 11305 import arsd.dom; 11306 } 11307 +/ 11308 version(linux) 11309 private extern(C) int eventfd (uint initval, int flags) nothrow @trusted @nogc; 11310 /* 11311 Copyright: Adam D. Ruppe, 2008 - 2022 11312 License: [http://www.boost.org/LICENSE_1_0.txt|Boost License 1.0]. 11313 Authors: Adam D. Ruppe 11314 11315 Copyright Adam D. Ruppe 2008 - 2022. 11316 Distributed under the Boost Software License, Version 1.0. 11317 (See accompanying file LICENSE_1_0.txt or copy at 11318 http://www.boost.org/LICENSE_1_0.txt) 11319 */