how to write systems paper.dvi

Transcript

1 An Evaluation the Ninth SOSP Submissions of r o (and Ho w Not) to W rite a Go o d System sP ap er Ho w Levin Ro D. Redell and y David Program Co-c hairmen SOSP Committee th Nin In tro duction 21, 1983, the program On for the 9th Symp osium on Op erating System Principles, Marc h committee read the eigh t y-three pap ers submitted, selected sixteen for presen tation at ha symp osium. ving the acceptance of ab out one in v e appro ximates those of past SOSPs, although the n um ber of This ratio eral as lo w er than in recen ty ears. Sev somewhat mem b ers of the program committee found submissions w surprisingly easy to separate the go o it pap ers from the bad ones; indeed, the ten committee mem b ers d quic agreed on the disp osition of o v er 80% of the pap ers. As the acceptance ratio indicates, most of kly w ere these rejections. mem committee completed its selection pro cess, the eral had b ers expressed disapp oin tmen t After sev the o v erall qualit y of the submissions. Ma y of the rejected pap ers exhibited similar w eaknesses, in raising eaknesses the committee felt should ha v e b een eviden t to the authors. In the hop e of that the w qualit of future SOSP submissions, and systems pap y generally , the committee decided to describ e ers the criteria used in ev aluating the pap ers it receiv ed. This article com bines the criteria used b y all of the mem ers of the committee, not just the author. b eha o a v oid sounding preac h y or p edagogic, w to v e cast this presen tation in the rst and second T try erson and adopted a ligh t, o ccasionally h umorous st yle. Nev ertheless, the in ten t is serious: to p oin t p the out problems that app ear rep eatedly in tec hnical pap ers in a w a y that will mak e it easier common then, for to a v oid them. As y ou read this article, authors supp ose y ourself to b e a prosp ectiv e future author for the 10th SOSP or for TOCS. Y ou'v e done some w ork y ou w ould lik e to publish, so y ou sit do wn write a pap er. What questions should y ou b e asking y ourself as y ou write? These are also the to asking that e, the review ers of y our pap er, will b e questions to determine its suitabilit y for publication. w Classes of P ap ers Y our pap er will probably fall naturally in to one of three categories: of  ts a real system, either b y a global surv ey presen an en tire system or b y a selectiv e examination It 35 Published in the A CM's Op er ating Systems R eview ,V olume 17, Num b er 3, July 1983 pages 35 { 40 Cop yrigh tA CM/SIGOPS

2 of sp themes em b o died in the system. eci c It presen a system that is unimplemen ted but utilizes ideas or tec hniques that y ou feel the  ts comm tec should kno w. unit hnical y delling a the theoretical areas, for example, p erformance mo in or securit yv eri-  addresses topic It cation. a single set of ev aluation criteria cannot b e applied uniformly across Ob categories; nev- viously , these y apply equally w ell to all three. As w e describ e criteria h one b elo w, w e will try to man ertheless, eac classes of pap ers to whic h it applies. Often it will emphasize e eviden t from con text. the b for aluation of Submissions Criteria Ev Original Ideas the ide as in the p ap er new? There is no p e t in submitting a pap er to a conference or journal A r oin original w ork unless the pap concerned con tains at least one new idea. with er do know? Y ou m ust b e famili ar with the you of the art and curren t researc h in the area How state v ered b yy our pap er in order co kno w that y our w ork is original. P erhaps the most common failing to among submissions in the rst category (real systems) w as an absence of new ideas; the systems the ed w frequen tly isomorphic to one of a small n um b er of pioneering systems w ell-do cumen ted describ ere literature. the in state the new ide ac oncisely? If y our pap er is to adv ance the state of kno wledge, y our reader Can you ry b able to nd the new ideas and understand them. T e writing eac h idea do wn in a paragraph ust m someone generally v ersed in the relev an t that can understand. If y ou can't, consider the p ossibilit y area that ou don't really understand the idea y ourself. When y ou ha v e the paragraphs, use them in the y for pap er. abstract the exactly is the pr oblem b eing solve d? Y our reader cannot b e exp ected to guess the What y ou problem faced en only a description of the solution. Be sp giv Be sure to explain wh yy our problem couldn't eci c. b e solv ed just as w ell b y previously published tec hniques. A r the ide as signi c ant enough to justify a p ap er? F requen tly , pap ers describing real systems con- e one or w o small enhancemen ts of established tec hniques. The new idea(s) can b e describ ed in a tain t a unnecessary w en t y-page pap er is paragraphs; and often obscures the actual inno v ation. Since few t author a system is a lot of w ork, the real of the pap er sometimes unconsciously confuses construction of total e ort with the w ork that is actually new. ("My team w ork ed on this system for t the oy ears and w w nally done. Let's tell the w orld ho ww onderful it is.") If the inno v ation is small, a small pap er or e're appropriate hnical a suitable journal is more in than an SOSP submission. tec note the work describ e d signi c antly di er ent fr om existing r Is d work? An ob vious extension to a elate previously algorithm, tec hnique, or system, do es published generally w arran t publication. Of course, not the lab el "ob vious" m ust b e applied carefully . (Remem b er the story of Colum bus demonstrating ho wto mak e egg stand on end (b y gen tly crushing it): "it's ob vious once I'v e sho wn y ou ho w".) Y ou m ust an w from y our w ork represen ts a signi can t departure sho the state of the art. If y ou can't, y ou should that y ask wh yy ou are writing the pap er and wh yan ourself one except y our mother should w an t to read it. y Is al l r elate d work r elate dfr om r efer enc es, and have you actual ly r e ad the cite d material? Y ou will sp ha dicult y con vincing the sk eptical reader of the originalit yof y our e orts unless y ou e eci cally v distinguish from previously published w ork. This requires it F urthermore, y ou will nd it citation. harder to con vince y our reader of the sup eriorit yofy our approac h if he has read the cited w orks and y ha v en't. ou 36 Published in the A CM's Op er ating Systems R eview ,V olume 17, Num b er 3, July 1983 pages 35 { 40 Cop yrigh tA CM/SIGOPS

3 A r omp arisons with pr evious work cle ar and explicit? Y ou cannot simply sa y: "Our approac h ec somewhat from adopted in the BagOfBits system [3]." Se sp eci c: "Our virtual memory di ers that approac managemen magnetic media rather than punc hed pap er tap e as in the BagOfBits system h t uses rate exp v emen ts in transfer impro and janitorial costs." the [3], with ected work c omprise a signi c ant extension, validation, or r epudiation of e arlier but unpr oven Do es the Implemen tation eriences supp orting or con tradicting a previously published pap er design ide as? exp are v w orth y candidates for publication. Designs and c heap, but implemen tations extremely are aluable based on unsound designs) are exp (particularly e. those ensiv is oldest p ap er you r efer enc e d? The newest? Have you r efer enc e d similar work at another What the d you enc e dte chnic al r ep orts, unpublishe efer memor anda, p ersonal c ommunic ations? Have r institution? ers to these questions help alert y ou to blind sp The in y our kno wledge or understanding. answ ots requen , pap ers with only v enerable references rep eat tly tly published w ork of whic h the author is F recen w are. P ap ers with only recen t references una "redisco v er" (through ignorance) old ideas. P ap ers often that only unpublished or unrefereed material tend to su er from narro wness and paro c hialism. cite b er citations not only ac kno wledge a debt to others, but also serv e as an abbreviation Remem that to If y our reader a complete dev elopmen t from rst principles. hanism the reader needs mec spare e some that dev acquire t, ho w ev er, he m ust b of able to con v ert y our citations in to source to elopmen he can read. P ersonal comm unicatio ns and in ternal memoranda fail this test. T ec hnical rep orts material frequen tities, published in limited quan are out-of-prin t, and dicult to obtain. Consequen tly , suc h tly wherev as should b e a v oided material er p ossible. citations source y Realit es the p ap er describ e something that has actual ly b e en implemente d? Quite a few of the SOSP Do pro ceeded for fteen pages in the presen t tense b efore rev ealing, in a concluding section (if submissions othetical all), the foregoing description w as of a h yp at system for whic h implemen tati on w as just that b eginning or b eing con templated. This is unacceptable. Y our reader has a righ t to kno w at the outset whether the under discussion is real or not. system the system b e en implemente d, how has it b e en use d, and what has this usage shown ab out the If has al of ortanc e of the ide as? Once again, a m ultiple man-y ear implemen tation e ort do es not actic pr imp publication con a pap er. If the implemen ted system justify tains new ideas, it is imp ortan tto itself of d w w ork ed out in practice. A seemingly go o ho idea that didn't pan out is at least as explain they teresting as one that did. It is imp ortan t to in e sp eci c and precise. "Our w eather prediction system b is and running and no one has complained ab out its o ccasional inaccurate forecasts" is m uc h less up w vincing erytime w e fail to forecast rain, the users hang their "ev et shirts o v er the tap e driv es con than dry". In the latter case, at least w e kno to that p eople are using and dep ending on the system. w If system hasn 't b e en implemente d, do the the as justify public ation now? This can b e a dicult ide question for an author to answ er dispassionately y et an y review er of the pap er will mak e this judgmen t. It is a ys tempting to write a design pap er describing a new system, then follo witupinay ear alw ers t with an "exp erience" pap er. The successful pap o of this genre nearly alw a ys include initial or w erience in the closing sections of the design pap er. The subsequen t exp erience pap er then deals exp with the learned from longer-term use of the system, frequen tly in unan ticipated w a ys. Review ers are lessons new v eptical of design-only pap ers unless there are sk ideas of ob viously high qualit y . ery Lessons What have you le arne dfr om the work? If y ou didn't learn an ything, it is a reasonable b et that y our and readers either, and y ou'v e simply w asted their time on't a few trees b y publishing y our pap er. w What should the r e ader le arn fr om the p ap er? Sp ell out the lessons clearly . Man y p eople rep eat the 37 17, in the A CM's Op er ating Systems R eview ,V olume Published Num b er 3, July 1983 pages 35 { 40 Cop yrigh tA CM/SIGOPS

4 mistak es history b ecause they didn't understand the history b o ok. of gener al applic able ar e these lessons? Be sure to state clearly the assumptions on whic hy our How ly Be conclusions generalizations based on lac kofkno wledge or exp erience. A particularly careful rest. of "real assuming pap ers is generalization from a single example, e.g., in that all common problem system" single are directories b y storing the directory in a implemen le and searc hing it linearly . system le ted y our conclusions, it helps to state the assumptions again. The reader ma y not ha v e seen When stating fteen them and ma yha v e forgotten them. Y ou ma yha v e also. for pages Choices wer the alternatives c onsider e d at various p oints, and why wer e the What es made the way they choic e elling A o d pap er do esn't just describ e, it explains. T e? y our readers what y ou did do esn't giv e wer go an y idea ho w carefully considered y our c hoices w ere. Y ou w an tto sa v e future researc hers from them wing the same blind alleys. Y ou also w an t to record p oten tially in teresting side-streets y ou didn't follo en to Mak e sure to state clearly whic h is whic h. happ explore. choic asons turn out to b e right, and, if so, was it for the r e the that motivate d them in the Did es exp e? not, what lessons ha v ey ou learned from the If erience? Ho w often ha v ey ou found rst plac ourself sa ying "this w orks, but for the wrong reason"? Suc h a pronouncemen t represen ts wisdom (at y least a amoun t) that ma y b ene t y our reader. Man y pap ers presen t a rational argumen t from small assumptions yan the w a y to the nished result when, in fact, the result w as obtained b initial en tirely all t fashioned and the deductiv e argumen t di eren later. This kind of "revisionist history" b orders on path ho dishonest prev en ts y our readers from understanding and w researc h really w orks. y Con text What ar e the assumptions on which the work is b ase d? The sk eptical reader is unlik ely to accept y our argumen ts their premises are stated. Mak e sure y ou get them all; it's easy to o v erlo ok implicit unless assumptions. A they r e alistic? F or "unimplem en ted systems" pap ers, this amoun ts to asking whether the as- r e the successful can hop e to supp ort a of implemen tatio n. Man y pap er designs are sumptions design treat e the real c haracteristics of comp onen ts they out abstractly , e.g., comm unication net w orks naiv ab h umans t yping on terminals. F or theoretical studies, it m ust b or clear ho w the assumptions re ect e realit , e.g., failure mo des in reliabilit y mo delling, classes of securit y threats in securit yv eri cation, y al in queuing systems. arriv distributions sensitive is the work to p ermutations of these assumptions? If y our result is delicately p oised How to a to w er of fragile assumptions, it will b e less useful on a reader than one that rests on a broader tall and rmer foundation. If a formal mo del is pr esente d, do es it give new information and insights? Simply de ning a mo del for its wn sak e is not v ery useful. One deep theorem is w orth a thousand de nitions. o o F cus es the intr o ductory material c ontain exc ess b aggage not ne e de d for you main development? "Real Do pap an are particularly guilt y of irrelev system" t description. If y our sub ject is distributed le sys- ers een tems, ysical c haracteristics of the connection b et w ph computer and comm unicatio n net w ork are the probably not germane. Av oid the temptation to describ e all ma jor c haracteristics of y our system at the will same of depth. Concen trate instead on the no v el or un usual ones that (presumably) el b e the lev fo of the original tec hnical cus ten t of the pap er. con Do you include just enough material fr om pr eviously publishe d works to enable your r e ader to fol low 38 17, in the A CM's Op er ating Systems R eview ,V olume Published Num b er 3, July 1983 pages 35 { 40 Cop yrigh tA CM/SIGOPS

5 your thr ad of ar gument? Do not assume that the reader has read ev ery referenced pap er within the e w eek has them at his ngertips for instan t reference. If y ou w an ty our reader to get past page last and v three, tro ductory sen tences of the form "W e adopt the de nition of transactions from Bro wn [4], oid a in it the to les as describ ed b y Green [7, 18], with ering notions of record and database in tro duced la y on y Blac and White [12] and later mo di ed b y Gra [10] [6]". On the other hand, don't burden y our y b k with length y extracts or paraphrases from cited w orks. reader unnecessarily Presen tation e the ide as or ganize d and pr esente dinacle ar and lo gic al way? A r ar e db efor e they de ne e use d? r terms A e forwar dr efer enc es kept to a minimum? Readers get anno y ed when they A eatedly encoun ter r rep ts e "Eac h le consists of a sequence of items, whic lik will b e describ ed in detail in a later statemen h The reader has to remem b er the tec hnical term section. but the term has no seman tics y et. "item", It's righ t to ask him to do this once or t wice, but only when absolutely necessary . Ev en if y ou all a ord the to explain "item" at this p oin t, giv e the reader enough information to attac h can't digression to a term: "Eac h le consists of meaning sequence of items, v ariable-sized, self-iden tifying bit some the w detailed terpretation will b e discussed b elo in under 'Multi-media Files'." Y our reader sequences whose y not y et understand y our concept of les completely , but at least he has some glimpse of the direction ma whic ou in are leading him. hy alternate or b e en c onsider e d? Theoretical pap ers, particularly of a mathematical Have ganizations describing are easier to organize than pap ers generally systems. The exp ected sequence of c haracter, lemma, theorem, example, corollary de nition, orks w ell for deductiv e argumen t, but p o orly for de- w scription. "real system" pap ers, m uc h dep ends on In in ten t: global surv ey or selectiv e treatmen t. the F requen tly , diculties in organization result from the author's un willingness to commit to either ap- proac h. whether y ou are surv eying y our system or fo cusing on a sp eci c asp ect and structure Decide pap er . the accordingly ide abstr rst? Do es it c ommunic ate the imp ortant written as of the p ap er? Abstracts in an act Was describing systems are sorely abused. The abstract is more often pap prose table of con ten ts than ers a precis the tec hnical con ten t of the pap er. of tends to come out something lik e this: "A system a It on Keysw orth's conceptualization of user in teraction [4] has b een designed and implem en based ted. Some results are presen ted and directions for future w ork considered." No reader skimmi ng preliminary the journal ely to k eep reading after that. Av oid lik passiv ev oice (despite tradition) and include a a is statemen t of assumptions and results. "W e designed and implem en ted simple user in terface follo wing a the of Keysw orth and disco v ered that con v erting the ideas bar to a to e p edal increases sp eed b y space TM Ho w ev er, accuracy decreased dramatically when w e pip ed ro c km usic instead of Muzak in 15%. to the oce." v e discussion and argumen t for the pap er. It helps to write the abstract b efore the pap er Lea the tradition) en the outline, since it fo cuses y our atten tion on ev main ideas y ou w an ts to (despite and v ey . con the p ap er nishe d? Review ers can often help y Is to impro v ey our pap er, but they can't write it ou for ou. Moreo v er, they can't b e exp ected to in y olate in sections mark ed "to b e included in the nal terp draft". In a mathematical pap er, a review er regards the statemen t of a theorem without pro of with suspicion, and, the theorem is in tended to culminate prior dev elopmen t, with in tolerance. Similarly , if tolerate in er describing a system, a review er cannot pap the omission of imp ortan t explanation or a justi cation. Omitting sections with a promise to ll them in later is generally unacceptable. W riting yle St the writing Is ar and c oncise? cle 39 Published in the A CM's Op er ating Systems R eview ,V olume 17, Num b er 3, July 1983 pages 35 { 40 Cop yrigh tA CM/SIGOPS

6 A r wor ds sp el le d and use dc orr e ctly? e r e sentenc es c omplete and gr ammatic al ly c orr e ct? A the e slang, and cuteness avoide d? r ambiguity, A ou don't ha v e sucien t concern for y our material to correct errors in gramma r, sp If and usage y elling, efore it for publication, Wh y should y ou exp ect submitting review er to read the pap er carefully? Some b a ers feel that this kind of carelessness is unlik ely to b review con ned to the presen tation, and will reject e the er at the rst inkling of tec hnical incoherence. Remem b er that y ou are asking a fa v or of y our pap ers: v let me con vince y ou that I ha review e done in teresting, publishable w ork." A review er is "Please receiv more orably disp osed to w ard y ou if he v es a clean, clear, carefully corrected man uscript than if it fa arriv es on o dd-sized pap er after ten trips through a photo copier and lo oking lik eitw as comp osed b ya grade-sc ho drop out. Ev en if y ou aren't particularly concerned with precise exp osition, there is certain ol to b in y our organization who is. Giv ey our man uscript someone this conscien tious soul and heed to e resulting suggestions. the Summary thirt y-o dd questions can help y ou write These b etter tec hnical pap er. Consult them often as y ou a organize our presen tation, write y our rst draft, and y y our man uscript in to its nal form. Some re ne of these questions address sp eci c problems in "systems" pap ers: others apply to tec hnical pap ers in y general. a go o d pap er is hard w ork, but riting ou will b e rew arded b y a broader distribution and W greater understanding of y our ideas within the comm unit y of journal and pro ceedings readers. 40 17, in the A CM's Op er ating Systems R eview ,V olume Published Num b er 3, July 1983 pages 35 { 40 Cop yrigh tA CM/SIGOPS

Related documents

Numerical Recipes

Numerical Recipes

Sample page from NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (ISBN 0-521-43108-5) Permission is granted for internet users to make one paper copy for their own personal use. Further reprod...

More info »
vol9 organic ligands

vol9 organic ligands

C HERMODYNAMICS HEMICAL T OMPOUNDS AND C OMPLEXES OF OF C U, Np, Pu, Am, Tc, Se, Ni and Zr O ELECTED WITH RGANIC L IGANDS S Wolfgang Hummel (Chairman) Laboratory for Waste Management Paul Scherrer Ins...

More info »
chemical table

chemical table

UPS Chemical Table - 49 CFR Version (Ground and Air Packages) GROUND AND AIR GROUND SHIPMENTS SHIPMENTS AIR SHIPMENTS BASIC DESCRIPTION FOR GROUND AND AIR Symbols: "‡" Requires a Technical Name / "*" ...

More info »
2017 NAICS Manual

2017 NAICS Manual

ORTH N A MERICAN I NDUSTRY LASSIFICATION C YSTEM S United States, 2017 EXECUTIVE OFFICE OF THE PRESIDENT AND BUDGET OFFICE OF MANAGEMENT

More info »
Little science, big science   and beyond

Little science, big science and beyond

LITTLE SCIENCE, BIG SCIENCE .AND . . BEYOND Derek J. de Price Sol la

More info »
ERG2016

ERG2016

2016 A guidebook intended for use by first responders A guidebook intended for use by first responders transportation incident during the initial phase of a during the initial phase of a transportatio...

More info »
How to Write a Good Scientific Paper

How to Write a Good Scientific Paper

How to Write a Good Scientific Paper Chris A. Mack

More info »
Technical Line: Lessee model comes together as leases project progresses

Technical Line: Lessee model comes together as leases project progresses

Applying IFRS IFRS 15 Revenue from Contracts with Customers A closer look at IFRS 15, the revenue recognition standard (Updated October 2018)

More info »
Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation

Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation

MANAGING THE RISKS OF EXTREME EVENTS AND DISASTERS TO ADVANCE CLIMATE CHANGE ADAPTATION SPECIAL REPORT OF THE INTERGOVERNMENTAL PANEL ON CLIMATE CHANGE

More info »
PB

PB

OFFICIAL 2019 CONNECTICUT PRACTICE BOOK (Revision of 1998) CONTAINING RULES OF PROFESSIONAL CONDUCT CODE OF JUDICIAL CONDUCT RULES FOR THE SUPERIOR COURT RULES OF APPELLATE PROCEDURE APPENDIX OF FORMS...

More info »
435 441 458 467r e

435 441 458 467r e

WT/DS435/R, WT/DS441/R WT/DS458/R, WT/DS467/R 28 June 2018 Page: (18 - 1/884 4061 ) Original: English AUSTRALIA CERTAIN MEASURES CON CERNING TRADEMARKS, – PACKAGING IONS AND OTHER PLAIN GEOGRAPHICAL I...

More info »
Why Forests? Why Now? The Science, Economics and Politics of Tropical Forests and Climate Change

Why Forests? Why Now? The Science, Economics and Politics of Tropical Forests and Climate Change

WHY FORESTS? WHY NOW? The Science, Economics and Politics of Tropical Forests and Climate Change Frances Seymour Jonah Busch

More info »
PowerPoint Presentation

PowerPoint Presentation

, ! h d c r te i a n c i e s t i d e r t e e h d g s i o l d o b n g u a p o t d D i t o D t e g w [email protected] Riverside, CA 92521 o IGK S Eamonn Keogh H University of California - Riverside Com...

More info »
Ludovico, Alessandro   Post Digital Print. The Mutation of Publishing Since 1894

Ludovico, Alessandro Post Digital Print. The Mutation of Publishing Since 1894

Post- Digital Print The Mutation of Publishing since 1894 Alessandro Ludovico OnOmat Opee 77

More info »
2018 Report Book 10 29.indb

2018 Report Book 10 29.indb

THE EXPANDING NEWS DESERT PENELOPE MUSE ABERNATHY Knight Chair in Journalism and Digital Media Economics The Expanding News Desert | 1

More info »
What Works in Girls' Education: Evidence for the World's Best Investment

What Works in Girls' Education: Evidence for the World's Best Investment

Foreword by Malala Yousafzai With a foreword by Winthrop s Malala Yousafzai p E Student, Nobel Peace Prize Laureate, and Cofounder of the Malala Fund rlin in What Works G Hard-headed evidence demonstr...

More info »
CBEC Electoral Board Decisions

CBEC Electoral Board Decisions

BOARD OF ELECTION COMMISSIONERS THE CITY OF CHICAGO FOR INDEX OF ELECTORAL BOARD DECISIONS April 201 4

More info »
Computer Vision: Algorithms and Applications

Computer Vision: Algorithms and Applications

Computer Vision: Algorithms and Applications Richard Szeliski September 3, 2010 draft c © 2010 Springer This electronic draft is for non-commercial personal use only, and may not be posted or re-distr...

More info »
pguo PhD grind

pguo PhD grind

T P H .D. G RIND HE A Ph.D. Student Memoir Philip J. Guo [email protected] Third Anniversary Reprint with margin notes from the perspective of a first-year assistant professor This book is free, but...

More info »