-
Notifications
You must be signed in to change notification settings - Fork 1.6k
RFC: Procedural macros in same package as app #3826
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 5 commits
093a48a
6400116
bd3762d
2abc88c
cf956c9
3ce11c2
269863b
001a769
a4cc177
dad0bef
f4ef425
6e1111a
3b34d68
2a98d11
9fd805f
4731ce1
1488bdb
904a687
2f47429
1dc76b1
cfb4c32
c7f34c5
3f94b5b
ff89040
ad1964f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
- Feature Name: `proc-macro-in-same-package-as-app` | ||
- Start Date: 2025-05-30 | ||
- RFC PR: [rust-lang/rfcs#3826](https://github.com/rust-lang/rfcs/pull/3826) | ||
- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) tbd | ||
|
||
# Summary | ||
[summary]: #summary | ||
|
||
Have a new folder in a cargo project, called `proc-macro`. This would be like the `tests` directory in that it is alongside the source code. It would eliminate the need to create an extra package for proc macros. | ||
|
||
# Motivation | ||
[motivation]: #motivation | ||
|
||
A common thing to ask about proc macros when first learning them is: "Why on earth does it have to be in a separate package?!" Of course, we eventually get to know that the reason is that proc macros are basically *compiler plugins*, meaning that they have to be compiled first, before the main code is compiled. So in summary, one needs to be compiled before the other. | ||
|
||
It doesn't have to be this way though, because we already have this mechanism of compiling one thing before another – for example, the `tests` directory. It relies on the `src` directory being built first, and likewise we could introduce a `proc-macro` directory that would compile before `src`. | ||
|
||
**To be absolutely clear**, this is not a proposal for same-*crate* proc macros (unlike previous proposals), but same-*package* proc macros: a much simpler problem. | ||
|
||
The motivation of this new directory comes down to just convenience. This may sound crude at first, but convenience is a key part of any feature in software. It is known in UX design that every feature has an *interaction cost*: how much effort do I need to put in to use the feature? For example, a feature with low interaction cost, with text editor support, is renaming a variable. Just press F2 and type in a new name. What this provides is incredibly useful – without it, having a subpar variable/function name needed a high interaction cost, especially if it is used across multiple files, and as a result, we are discouraged to change variable names to make it better, when we have retrospect. With a lower interaction cost, the renaming operation is greatly promoted, and leads to better code. | ||
|
||
This proposal aims smooth out the user experience when it comes to creating new proc macro, and achieve a similar effect to the F2 operation. It is important to emphasise that proc macros can dramatically simplify code, especially derive macros, but they a lot of the times aren't used because of all the extra hoops one has to get through. This would make proc macros (more of) "yet another feature", rather than a daunting one. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm missing a clear statement which proc-macros is this feature for? The motivation section focuses on the ease of introducing a proc macro in a library that currently doesn't have one. I have sympathy for that, the proc-macro <-> build script/decl-macro switcheroo isn't fun. If the feature was tightly scoped to experimentation and simple library-internal use, the build order questions discussed in #3826 (comment) have a clear, simple answer. Dependents also wouldn't need to know that the proc macro exists, let alone opt in/out with crate features (cc #3826 (comment)). But it seems quite a few people commenting here see this feature as a general replacement for "library re-exports proc-macros" -- and it's not like we can stop people from re-exporting the proc macros like that, even if that wasn't the intended usage. If the feature isn't designed with that in mind then there's a risk it'll end up as a newbie trap: easier to get started, but once you get serious you have to dig yourself back out of the downsides of the easier way. But in that case, the feature will become much more complicated. Thus, my questions: Is it an explicit design goal that essentially any library providing proc macros could use this feature? If not, what's out of scope and how would people pick between using this feature and making a separate crate? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What do you see as the downsides to designing a proc-macro in this way? I don't think I've seen those raised yet in this discussion. I've seen this as a general way of doing proc-macros and would see this feature not to be worth it (or maybe actively opposed) if this is for smaller scope and then people grow out of it. In general, I'm not thrilled with cramming everything into a single package to avoid having a workspace and feel that to meet what people want in doing that would move cargo away from what its good for. That said, I feel like this doesn't run into those problems and can help avoid some proc-macro problems (e.g. the proc-macro logically depends on the library it re-exports but can't declare that relationship nor is it obvious of a problem for users to solve or how to solve it). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. FWIW, I share your gut feeling that this feature probably isn't worth it if it's only useful for small, simple macros, but I didn't want to prejudice my comment with that. The downsides I see with trying to do "everything" are mostly the complexity that I expect will be required to address all of the concerns raised in other comment threads. To avoid introducing downsides compared to a separate package, this feature will need:
All of these points (and others we may find) can be addressed in theory. But not all of them seem to have straightforward solutions that fit in the scope of what a package currently is. For example, something like
This is a problem worth solving, but I'd prefer a solution that existing libraries with separate There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe there is something I'm missing but I feel like the answers to these questions are fairly simple; the author just needs to take the time to enumerate and write it up. For example:
This is no different than the regular proc-macro situation. If you are like
This is why I brought up [proc-macro]
required-features = ["derive"] And with that, your proc-macro is now skipped if You mentioned a problem with this related to addressing one of N libraries in a package. That didn't quite make sense to me but if its related to the other points, then I addressed those.
While true that if everyone depended on
That proposal has a lot of complexity to it that I do not see it being likely to happen. This introduces cycles between packages, particularly for publishing. There are ways to solve problems with that but taking just one part as an example, this would require a new publish API which has been stalled out for years. This also takes a concern I have with this proposal being able to solve There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The difference is that with separate packages, it's clear how any of the three options can expressed. The library either depends on the associated proc-macro package or doesn't; a facade crate like As I've said, I'm sure there's many ways to design support for this. But without having seen a complete proposal for all aspects, I'm no sure whether the end result will truly be better than the status quo.
It's not a theoretical benefit. Inertia is strong, of course, but I've seen a few big dependency trees reach this goal. For example, most configurations of Besides, I don't understand how a clap-style (or There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It wasn't an initial goal, but in the pre-RFC and RFC there has been a need to do it so a lot of the revisions do have that in mind. There is still a lot of room of iterations.
You can just use conditional compilation on your proc macros. It'll just be like how you gate all other features. Unless I'm overlooking something. It feels that some of the criticisms, while valid, are a bit too fixated on the idea of the separation between proc-macro packages and library packages. There was really no reason for them to be separated in the first place, except for build system, which I think most of us can agree is an unfortunate blocker. In an ideal world, proc macros and code can coexist together, and this is a close equivalent. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I fundamentally disagree with the motivation of the RFC. If that motivation was the sole motivation, I would be a hard "no" on this. imo if people have a problem with managing a workspace, we should work to improve workspaces. I mentioned earlier in this thread (#3826 (comment)) that I generally disagree with cramming everything into a package. For more on this topic, see https://blog.rust-lang.org/inside-rust/2024/02/13/this-development-cycle-in-cargo-1-77/#when-to-use-packages-or-workspaces As I mentioned before (#3826 (comment)), the main benefit this gives us is a happy path for declaring the relationship between the proc-macro and the library it generates code against. As a secondary motivation is I see this as a fundamental step towards unblocking So for
I think there is something missing in our communication because you seem to still feel there are significant open questions on this point while I feel that I explain how this naturally falls out of the design and there isn't anything special to this (though the RFC does need to explicitly call this all out). If there are lingering concerns, feel free to reach out to me for a call for us to work this out.
Ok, so I take back my saying its theoretical. I still think it is not a workflow we should prioritize in our designs. Cargo is an opinionated tool focused on usability. Having to do that much precise coordination across a dependency tree seems like the wrong answer with Cargo's design. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thank you for elaborating on your perspective on what the RFC should be, @epage, this helps me understand your previous comments better. Also thanks for offering a call, but I think at this point it's probably a better use of everyone's time to let the RFC catch up with the discussion and propose a more detailed design. Then I'll open more focused comment threads if I still have concerns, because then I'll be able to phrase them more precisely since I'll have less guessing to do about what precisely is proposed. The one thing that I'd like to ask you which the updated RFC probably won't answer is about your plan to merge @ora-0 I'd say I care more about the build system POV (which units of work exist in the build and how they depend on each other) than about packages per se. In essence, if the goal is that every proc macro should share a package with its associated library, then I'm asking what that means for the dependency graph at the build system level. If some useful dependency graphs would be inexpressible in that future style, then I'd consider that a downside of the RFC and would like to discuss the impact. Conversely, if the RFC doesn't restrict our ability to massage the work done by the build system in the most favorable shape (compared to the status quo), I'd like the RFC to spell out how this is possible. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There are two common dependency problems when you have a
Since
|
||
|
||
An objection to this one might raise is "How much harder is typing in `cargo new` than `mkdir proc-macro`?" But we should consider if we would still use as much integration tests if the `tests` directory if it is required to be in a seperate package. The answer is most likely less. This is because (1) having a new package requires ceremony, like putting in a new dependency in cargo.toml, and (2) requires adding to the project structure. A *tiny* bit in lowering the interaction cost, even from 2 steps to 1, can greatly improve the user experience. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMHO, a greater motivation is to avoid having to make several upload to crates.io and making sure to keep them in sync. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For me, its less of an issue publishing multiple crates but getting the pattern right. In general, there is a conflict in people wanting to treat a package as a whole workspace the the format complexity that doing so entails (details). I'm generally in favor of encouraging splitting packages. proc-macros are odd because they have a logical dependency on the library that re-exports them because they generate code that calls into that library. Its easy to overlook this problem and there isn't a great way to declare this relationship today. The options are
If this also helps towards |
||
|
||
In summary (TL;DR), the effort one needs to put in to use a feature is extremely important. Proc macros currently has a higher ceiling, needing one to create a whole new package in order to use it, and lowering the ceiling, even just a little bit, could massively improve user experience. This proposal can lower it. | ||
|
||
# Explanation | ||
[explanation]: #explanation | ||
epage marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It would be good to step through a couple of well known or exotic proc-macros, showing what this would be like. For example, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When looking at those proc macros, another area to consider is how they do testing and how that maps into this new design |
||
|
||
Currently, we create a new proc macro as so: | ||
1. Create a new package | ||
2. In its cargo.toml, specify that it is a proc macro package | ||
3. In the main project, add the package as a dependency | ||
4. Implement the proc macro in the new package | ||
epage marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
After this change, we create a new proc macro like this: | ||
1. Create a new directory called `proc-macro` alongside your `src` directory | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Personally, I am not convinced that it should be a directory. I think it could be a single Also, while having convention is great, I think it should be possible to reference explicitly the files in Cargo.toml, just like we have There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I suspect enough proc-macros are bigger than what would be appropriate for a single file We should also consider a full There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There would also need to be a way to turn off auto-discovery like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could also check for either There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We allow that mixing of file or directory with bins, examples, and tests, so there is precedence for it. Just nothing at the top-level atm. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I'm not sure how this would work, since it would depend on the main code being built first, which it can't. Edit: Ah, I see #3826 (comment) |
||
2. Implement the proc macro in a new file in `proc-macro`. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What file inside the directory? Is it There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I chose There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How exactly does this work - would Personally I think the second option is better so it's less confusing to share code among multiple macros. Whatever the decision, the alternative should be mentioned in "rationale and alternatives". |
||
|
||
To use the proc macro, simply import it via `crate::proc_macro`. | ||
```rust | ||
use crate::proc_macro::my_file::my_macro; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The interactions with Anything more magic then that needs @ora-0 these exact details need to get discussed somewhere, i.e. "how does one reproduce this behavior with two separate crates and the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (just seeing this is effectively @kennytm's comment at #3826 (comment)) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree that this should avoid all rustc/language changes and be Cargo-only. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think having the extern crate be named There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think minor rustc alterations should be out of the question but obviously I would want a weigh-in from @petrochenkov on any proposals involving preludes or special resolution and I imagine he will prefer a strong justification. |
||
``` | ||
Or, if the file happens to be `mod.rs`, you can access it directly after the `proc_macro` bit. | ||
|
||
## Proc Macro Libraries | ||
Libraries like `syn`, `quote`, and `proc-macro2`, would be included under `[dev-dependecies]` in the cargo.toml. (Perhaps we should put it in build dependencies? or a new dependency section for proc macros.) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It would have to be in Another option would be to use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Complexity isn't the main concern but it has a major impact on the ecosystem that would need to be worth the cost for all of the relevant tools to be updated to process a new dependency table |
||
|
||
## How it would work in the implementation | ||
Cargo would have to compile the `proc-macro` directory first, as a proc macro type (of course). Then, in compiling the main code, `crate::proc_macro::file_name::my_macro` would resolve the module to the file `proc-macro/file_name.rs`. Alternatively, if the user uses `mod.rs`, it would be resolved from `crate::proc_macro::my_macro`. This would finally be passed into rustc. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ogoffart from #3826 (review)
(split in a thread to make it not lost) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It should also be specified whether it should be built before or in parallel with the main crate, i.e. if the proc macro should be available to the main crate. In some cases (e.g. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For some code, the main crate depends on the proc-macro so it can both export it and use it internally, e.g.: fayalite There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It has to be built before the main crate because it is built before lib. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
IMO it should be configurable then. Having this dependency by default when not necessary and reducing pipelining (and hence increasing compile times) would be unfortunate.
What are "main crate" and "lib" for you? They are the same for me. And even if they were different, why does the proc macro need to be built before the lib? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we have any future possibilities that rustc allows building, for example, most of serde except the parts that rely on There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume you are referring to the workaround of people depending on imo the correct way of solving that is to pull out a This is discussed more in the thread #3826 (comment) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @ogoffart from #3826 (review)
(split in a thread to make it not lost) |
||
|
||
# Drawbacks | ||
[drawbacks]: #drawbacks | ||
|
||
1. The proc macro directory cannot use functions from src. (but that was not possible before anyways) | ||
|
||
# Rationale and alternatives | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Like @ogoffart pointed out, this isn't new. It would be good to step through past discussions and compare to where those landed There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you go into the motivation for This is an ineresting one because build scripts do not have a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's because when one types There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That doesn't quite make sense to me. We could have a Whether to have the flag or not can also be affected by whether we support unit tests inside of it (we don't for build scripts) so you can run Note: even if nothing is changed from this, this fleshes out the rationale and should be included / summarized in the RFC. Side note: whats interesting is we're working on moving build scripts out into their own packages while this does the opposite for proc macros. I think the use cases are different enough that one does not affect the other but kind of interesting to observe. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should talk through naming
Why aren't we consistent? Why have the name in one direction or the other? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's true. Do we prefer There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would lean towards There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you add a discussion under the Rationale and Alternatives that speaks to what was considered for naming and why the current option was chosen? |
||
[rationale-and-alternatives]: #rationale-and-alternatives | ||
|
||
> Have proc macro files marked `#![proc_macro_file]` to signal to cargo to compile it first. | ||
|
||
Since it would compile first, proc macro files cannot import functions in the main code. The problem is having it side-by-side to the rest of your code makes it seem like you could just import it, when you cannot. Having it as a seperate directory makes clear of this. | ||
|
||
> Eliminate the need for new proc macro files/folders entirely, have the compiler work out where the proc macros are and separate them. | ||
|
||
This would suffer from the same issue as the last alternative, plus being harder to implement. | ||
|
||
> Introspection | ||
|
||
Harder to implement, with less payoff relative to the amount of work required. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this for metaprogramming with reflection? In that case, another alternative is the work to make declarative macros cover more use cases of proc-macros. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since this RFC is mostly a build system change, the alternatives should also be mostly related to the build system. Everything currently in this section would be on the rustc / lang side. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Eh, this is a change trying to solve a problem for users. There are multiple ways of solving that problem, build system or not. In general, I think we need to be thinking cross-team more in how we solve problems. |
||
|
||
# Prior art | ||
[prior-art]: #prior-art | ||
|
||
1. Zig comptime: metaprogramming code can sit directly next to application code. | ||
2. Declarative macros: can sit side by side as well, but is less powerful. | ||
3. Lisp macros: same as last two, except more powerful. | ||
4. `tests` directory, and `build.rs`: compiled at a different time as the main code. | ||
5. `Makefiles`, or other build systems: they allow for customisability for when code is built. | ||
|
||
# Unresolved questions | ||
[unresolved-questions]: #unresolved-questions | ||
|
||
1. Should proc macro dependencies be listed under `[dev-dependencies]`, `[build-dependencies]`, or a new `[proc-macro-dependencies]` section? | ||
2. Should we import like `crate::proc_macro::file::macro`, or via a new keyword, like `crate_macros::file::macro`? The latter would avoid name collisions, but might be more confusing. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Without changing rustc itself or inserting some hidden prelude I don't see how can the proc-macro be available from Making it available from |
||
|
||
# Future possibilities | ||
[future-possibilities]: #future-possibilities | ||
|
||
As described in the [motivation] section, this proposal is aimed to make the process of creating proc macros easier. So a natural extension of this is to remove the need of third-party libraries like syn and proc-macro2. There is already an effort to implement quote, so they might be a possibility. | ||
|
||
Second, this might enable for some sort of `$crate` metavariable. | ||
|
||
Another possibility is possibly allowing for proc macro crates to export data structures, which would make writing certain things easier. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think
build.rs
is a much better example that the tests directory, because like proc macros, it needs to be build before the main crate.