What is VR/AR Development?

VR/AR development involves creating immersive digital experiences using:

• VR (Virtual Reality): Fully immersive, computer-generated environments where users are completely cut off from the real world (using headsets like Meta Quest, HTC Vive).

• AR (Augmented Reality): Overlays digital content onto the real world, enhancing reality rather than replacing it (using phones, tablets, or glasses like Microsoft HoloLens).

Key Development Platforms & Tools

1. Unity 6 (Latest Version - 2024)

Unity is the industry-leading game engine for VR/AR development, used in ~60% of all VR/AR projects.

Strengths for VR/AR:

• XR Interaction Toolkit: Official framework for building VR/AR interactions

• Universal Render Pipeline (URP): Optimized graphics for VR performance

• AR Foundation: Single API for building AR apps across iOS/Android

• Asset Store: Massive library of ready-made assets and tools

• Multi-platform support: Build for Quest, PC VR, PSVR, iOS, Android from one project

Common Use Cases:

• Enterprise training simulations

• Architectural visualization

• Interactive VR games

• Medical/surgical training

• Retail and product visualization

Development Languages: C# (primary), Unity Visual Scripting (Bolt)

2. Three.js

Three.js is a lightweight JavaScript 3D library for creating 3D experiences that run in web browsers.

Strengths for Web-Based VR/AR:

• WebXR API: Built-in support for browser-based VR/AR

• No installation needed: Users access via URL

• Easy to deploy: Standard web hosting

• Progressive enhancement: Can work on both desktop and mobile

• Large community: Extensive examples and documentation

Common Use Cases:

• Product configurators on e-commerce sites

• Educational web experiences

• Data visualization in 3D

• Simple interactive web games

• Marketing and brand experiences

Development Stack: JavaScript/TypeScript, HTML5, WebGL

Development Workflow Comparison

Essential VR/AR Development Concepts

Core Technical Challenges

1. Performance Optimization: Maintaining 90+ FPS for VR to prevent motion sickness

2. User Interaction: Designing intuitive 3D UI/UX (not just 2D interfaces in 3D space)

3. Comfort Design: Minimizing VR sickness through proper movement techniques

4. Spatial Audio: 3D sound that changes based on head movement

5. Hand/Gaze Tracking: Natural input methods (hands, eye tracking, voice)

Common Interaction Patterns

• Teleportation: Point-and-move to reduce motion sickness

• Direct Manipulation: Grabbing and moving objects with virtual hands

• Ray Interaction: Pointing at distant objects with laser pointers

• Gesture Recognition: Hand/finger tracking for natural interactions

• Voice Commands: Speech-to-text for hands-free control

Current Industry Trends (2024)

VR Development Focus:

• Standalone VR headsets: Meta Quest 3, Apple Vision Pro development

• Social VR: VRChat worlds, Meta Horizons, Rec Room rooms

• Fitness & Wellness: Beat Saber clones, meditation apps

• Enterprise training: Walmart, Boeing, etc. using VR for employee training

• Passthrough AR: Mixed reality on VR headsets

AR Development Focus:

• Mobile AR: Snapchat/Instagram filters, Pokémon GO-like experiences

• WebAR: No-app-needed AR via browser

• Markerless/World Tracking: AR that understands surfaces and objects

• Try-before-you-buy: Furniture, clothing, makeup visualization

Getting Started Paths

For Beginners:

1. Start with Unity + Oculus Integration (if interested in VR games/apps)

2. Start with Three.js + A-Frame (if coming from web development background)

3. Start with Spark AR/Instagram filters (for simple mobile AR)

Learning Resources:

• Unity Learn: Official Unity VR/AR tutorials

• Oculus Developer Center: Best practices for Quest development

• Three.js Journey: Paid but excellent Three.js course

• WebXR Samples: Google's collection of WebXR examples

• MIT Reality Hack: Annual hackathon for XR development

VR/AR Development: Three.js & Unity 6

1. VR/AR Fundamentals & Ecosystem

A. Understanding the Spectrum

text

┌─────────────────────────────────────────────────────────┐
│               Extended Reality (XR) Spectrum             │
├─────────────────────────────────────────────────────────┤
│ Virtual Reality (VR)         │ Augmented Reality (AR)   │
│ • Fully immersive            │ • Digital overlay on real│
│ • Headset required           │ • Phone/Glasses/HoloLens │
│ • 6DOF tracking              │ • Marker/Markerless      │
│ • Examples: Quest, Valve Index│ • Examples: ARKit, ARCore│
│                              │                          │
│ Mixed Reality (MR)           │ WebXR                    │
│ • Digital+physical interact  │ • Browser-based XR       │
│ • Spatial understanding      │ • Progressive Web Apps   │
│ • Examples: HoloLens, Magic  │ • Three.js + WebXR API   │
│   Leap                       │                          │
└─────────────────────────────────────────────────────────┘

B. Hardware Landscape 2024

C. Key Concepts

Degrees of Freedom (DOF):

• 3DOF: Rotation only (pitch, yaw, roll) - Cardboard, GearVR

• 6DOF: Rotation + Position (x, y, z) - Modern VR

• 7DOF: 6DOF + Scale - Advanced tracking

Field of View (FOV):

• Human: ~210° horizontal

• Quest 3: 110° horizontal

• Varjo XR-4: 115° horizontal

Refresh Rates & Latency:

• Minimum: 72Hz (Quest 1)

• Target: 90-120Hz (comfort)

• Critical: <20ms motion-to-photon latency

2. Three.js for WebXR Development

A. Three.js Architecture for XR

javascript

// Three.js XR Application Structure
class XRApplication {
    constructor() {
        // Core Three.js components
        this.scene = new THREE.Scene();
        this.camera = new THREE.PerspectiveCamera();
        this.renderer = new THREE.WebGLRenderer({
            antialias: true,
            alpha: true,
            powerPreference: 'high-performance'
        });
        
        // XR-specific systems
        this.xrManager = null;
        this.controllers = [];
        this.hitTestSource = null;
        this.referenceSpace = null;
        
        // Performance monitoring
        this.stats = new Stats();
        this.clock = new THREE.Clock();
        
        this.init();
    }
    
    async init() {
        // Setup renderer
        this.renderer.setPixelRatio(window.devicePixelRatio);
        this.renderer.setSize(window.innerWidth, window.innerHeight);
        this.renderer.xr.enabled = true;
        this.renderer.outputEncoding = THREE.sRGBEncoding;
        this.renderer.toneMapping = THREE.ACESFilmicToneMapping;
        
        // Setup scene
        this.setupLighting();
        this.setupEnvironment();
        
        // Initialize WebXR
        await this.initWebXR();
        
        // Start animation loop
        this.animate();
    }
    
    async initWebXR() {
        // Check WebXR support
        if (!navigator.xr) {
            this.showWebXRNotSupported();
            return;
        }
        
        // Request XR session
        const sessionInit = {
            optionalFeatures: [
                'local-floor',
                'bounded-floor',
                'hand-tracking',
                'hit-test',
                'anchors',
                'plane-detection',
                'depth-sensing',
                'layers'
            ],
            requiredFeatures: ['local-floor']
        };
        
        try {
            const session = await navigator.xr.requestSession('immersive-vr', sessionInit);
            
            // Setup Three.js XR manager
            this.xrManager = new THREE.WebXRManager(this.renderer, session);
            await this.xrManager.setSession(session);
            
            // Get reference space
            this.referenceSpace = await session.requestReferenceSpace('local-floor');
            
            // Setup controllers
            this.setupControllers(session);
            
            // Setup hit testing
            await this.setupHitTest(session);
            
        } catch (error) {
            console.error('WebXR session failed:', error);
        }
    }
    
    setupControllers(session) {
        // Create controller models
        const controllerModelFactory = new THREE.XRControllerModelFactory();
        
        for (let i = 0; i < 2; i++) {
            const controller = this.renderer.xr.getController(i);
            this.scene.add(controller);
            
            // Add visual representation
            const controllerGrip = this.renderer.xr.getControllerGrip(i);
            controllerGrip.add(controllerModelFactory.createControllerModel(controllerGrip));
            this.scene.add(controllerGrip);
            
            // Setup controller events
            controller.addEventListener('selectstart', this.onSelectStart.bind(this));
            controller.addEventListener('selectend', this.onSelectEnd.bind(this));
            controller.addEventListener('squeezestart', this.onSqueezeStart.bind(this));
            
            this.controllers.push(controller);
        }
        
        // Setup hand tracking if available
        if (session.enabledFeatures.includes('hand-tracking')) {
            this.setupHandTracking(session);
        }
    }
    
    async setupHitTest(session) {
        // Create hit test source for AR placement
        const viewerSpace = await session.requestReferenceSpace('viewer');
        this.hitTestSource = await session.requestHitTestSource({
            space: viewerSpace
        });
        
        // Create reticle for placement
        this.reticle = new THREE.Mesh(
            new THREE.RingGeometry(0.15, 0.2, 32),
            new THREE.MeshBasicMaterial({ color: 0xffffff })
        );
        this.reticle.matrixAutoUpdate = false;
        this.reticle.visible = false;
        this.scene.add(this.reticle);
    }
    
    setupHandTracking(session) {
        // Hand tracking setup
        this.handJoints = {};
        
        for (let i = 0; i < 2; i++) {
            const hand = this.renderer.xr.getHand(i);
            
            // Create joint visualization
            const handModel = new THREE.Mesh(
                new THREE.SphereGeometry(0.01),
                new THREE.MeshBasicMaterial({ color: 0x00ff00 })
            );
            hand.add(handModel);
            this.scene.add(hand);
            
            // Track joint positions
            hand.addEventListener('connected', (event) => {
                const xrInputSource = event.data;
                if (xrInputSource.hand) {
                    this.setupHandJoints(xrInputSource.hand, hand);
                }
            });
        }
    }
    
    setupHandJoints(xrHand, handObject) {
        // Map XR hand joints to Three.js objects
        const jointMeshes = {};
        
        for (const jointName in xrHand) {
            const jointMesh = new THREE.Mesh(
                new THREE.SphereGeometry(0.005),
                new THREE.MeshBasicMaterial({ color: 0xff0000 })
            );
            jointMeshes[jointName] = jointMesh;
            handObject.add(jointMesh);
        }
        
        this.handJoints[handObject.id] = jointMeshes;
    }
    
    animate() {
        this.renderer.setAnimationLoop((timestamp, frame) => {
            // Update stats
            this.stats.begin();
            
            // Update time
            const delta = this.clock.getDelta();
            
            // Update XR frame
            if (frame && this.xrManager) {
                this.updateXRFrame(frame);
            }
            
            // Update scene
            this.update(delta);
            
            // Render
            this.renderer.render(this.scene, this.camera);
            
            this.stats.end();
        });
    }
    
    updateXRFrame(frame) {
        // Process hit test results
        if (this.hitTestSource && frame) {
            const hitTestResults = frame.getHitTestResults(this.hitTestSource);
            
            if (hitTestResults.length > 0) {
                const hit = hitTestResults[0];
                const pose = hit.getPose(this.referenceSpace);
                
                if (pose) {
                    this.reticle.visible = true;
                    this.reticle.matrix.fromArray(pose.transform.matrix);
                }
            } else {
                this.reticle.visible = false;
            }
        }
        
        // Update hand joints
        this.updateHandJoints(frame);
    }
    
    updateHandJoints(frame) {
        for (const handId in this.handJoints) {
            const joints = this.handJoints[handId];
            const xrHand = frame.getHand(parseInt(handId));
            
            if (xrHand) {
                for (const jointName in joints) {
                    const joint = xrHand[jointName];
                    if (joint) {
                        const jointPose = frame.getJointPose(joint, this.referenceSpace);
                        if (jointPose) {
                            joints[jointName].position.set(
                                jointPose.transform.position.x,
                                jointPose.transform.position.y,
                                jointPose.transform.position.z
                            );
                        }
                    }
                }
            }
        }
    }
}
B. Advanced Three.js XR Features
javascript

// 1. Spatial Anchors & Persistence
class XRSpatialAnchors {
    constructor(xrManager, scene) {
        this.xrManager = xrManager;
        this.scene = scene;
        this.anchors = new Map();
        this.anchorMeshes = new Map();
    }
    
    async createAnchor(position, rotation) {
        const session = this.xrManager.getSession();
        
        // Create XR anchor
        const anchorPose = new XRRigidTransform(position, rotation);
        const anchor = await session.createAnchor(anchorPose, this.xrManager.referenceSpace);
        
        // Create visual representation
        const anchorMesh = new THREE.Mesh(
            new THREE.BoxGeometry(0.1, 0.1, 0.1),
            new THREE.MeshBasicMaterial({ color: 0xff0000 })
        );
        anchorMesh.position.copy(position);
        anchorMesh.quaternion.copy(rotation);
        this.scene.add(anchorMesh);
        
        // Store anchor
        this.anchors.set(anchor, { position, rotation });
        this.anchorMeshes.set(anchor, anchorMesh);
        
        return anchor;
    }
    
    async saveAnchors() {
        const anchorsData = [];
        
        for (const [anchor, data] of this.anchors) {
            anchorsData.push({
                uuid: anchor.uuid,
                position: data.position.toArray(),
                rotation: data.rotation.toArray()
            });
        }
        
        // Save to localStorage or server
        localStorage.setItem('xr_anchors', JSON.stringify(anchorsData));
    }
    
    async loadAnchors() {
        const savedAnchors = JSON.parse(localStorage.getItem('xr_anchors') || '[]');
        
        for (const anchorData of savedAnchors) {
            const position = new THREE.Vector3().fromArray(anchorData.position);
            const rotation = new THREE.Quaternion().fromArray(anchorData.rotation);
            
            await this.createAnchor(position, rotation);
        }
    }
}

// 2. Plane Detection & Occlusion
class XRPlaneDetection {
    constructor(xrManager, scene) {
        this.xrManager = xrManager;
        this.scene = scene;
        this.planes = new Map();
        this.planeMeshes = new Map();
        this.occlusionMaterial = new THREE.MeshBasicMaterial({
            colorWrite: false,
            depthWrite: true
        });
    }
    
    async enablePlaneDetection() {
        const session = this.xrManager.getSession();
        
        // Request plane detection
        await session.updateWorldTrackingState({
            planeDetectionState: { enabled: true }
        });
        
        // Listen for plane updates
        session.addEventListener('planesdetected', this.onPlanesDetected.bind(this));
    }
    
    onPlanesDetected(event) {
        const detectedPlanes = event.detectedPlanes;
        
        for (const plane of detectedPlanes) {
            if (!this.planes.has(plane)) {
                this.createPlaneVisualization(plane);
            } else {
                this.updatePlaneVisualization(plane);
            }
        }
        
        // Remove old planes
        for (const existingPlane of this.planes.keys()) {
            if (!detectedPlanes.includes(existingPlane)) {
                this.removePlaneVisualization(existingPlane);
            }
        }
    }
    
    createPlaneVisualization(plane) {
        // Get plane geometry
        const polygon = plane.polygon;
        const vertices = [];
        
        for (const vertex of polygon) {
            vertices.push(new THREE.Vector3(vertex.x, vertex.y, vertex.z));
        }
        
        // Create mesh
        const geometry = new THREE.BufferGeometry().setFromPoints(vertices);
        const mesh = new THREE.Mesh(
            geometry,
            new THREE.MeshBasicMaterial({
                color: 0x00ff00,
                transparent: true,
                opacity: 0.3,
                side: THREE.DoubleSide
            })
        );
        
        this.scene.add(mesh);
        this.planes.set(plane, mesh);
        this.planeMeshes.set(mesh, plane);
    }
    
    // 3. Depth Sensing for Occlusion
    async enableDepthSensing() {
        const session = this.xrManager.getSession();
        
        // Request depth sensing
        await session.updateWorldTrackingState({
            depthSensingState: {
                usagePreference: ['cpu-optimized', 'gpu-optimized'],
                dataFormatPreference: ['luminance-alpha', 'float32']
            }
        });
        
        // Create depth texture
        const depthData = session.depthData;
        if (depthData) {
            this.createDepthTexture(depthData);
        }
    }
    
    createDepthTexture(depthData) {
        const depthTexture = new THREE.DataTexture(
            depthData.data,
            depthData.width,
            depthData.height,
            THREE.RGBAFormat,
            THREE.FloatType
        );
        depthTexture.needsUpdate = true;
        
        // Apply to shader for occlusion
        const occlusionShader = {
            uniforms: {
                depthTexture: { value: depthTexture },
                cameraNear: { value: 0.1 },
                cameraFar: { value: 100 }
            },
            vertexShader: `
                varying vec2 vUv;
                void main() {
                    vUv = uv;
                    gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
                }
            `,
            fragmentShader: `
                uniform sampler2D depthTexture;
                uniform float cameraNear;
                uniform float cameraFar;
                varying vec2 vUv;
                
                float readDepth(vec2 coord) {
                    float depth = texture2D(depthTexture, coord).r;
                    return perspectiveDepthToViewZ(depth, cameraNear, cameraFar);
                }
                
                void main() {
                    float depth = readDepth(vUv);
                    // Perform occlusion testing
                    gl_FragColor = vec4(vec3(depth), 1.0);
                }
            `
        };
        
        this.occlusionMaterial = new THREE.ShaderMaterial(occlusionShader);
    }
}

// 4. Physics Integration for XR
class XRPhysics {
    constructor(scene, gravity = -9.81) {
        this.scene = scene;
        this.world = new CANNON.World();
        this.world.gravity.set(0, gravity, 0);
        this.world.broadphase = new CANNON.NaiveBroadphase();
        
        this.meshes = new Map();
        this.bodies = new Map();
        
        // Create ground plane for physics
        this.createGround();
    }
    
    createGround() {
        // Cannon.js ground body
        const groundShape = new CANNON.Plane();
        const groundBody = new CANNON.Body({ mass: 0 });
        groundBody.addShape(groundShape);
        groundBody.quaternion.setFromAxisAngle(
            new CANNON.Vec3(1, 0, 0),
            -Math.PI / 2
        );
        this.world.addBody(groundBody);
        
        // Three.js ground mesh
        const groundGeometry = new THREE.PlaneGeometry(100, 100);
        const groundMaterial = new THREE.MeshStandardMaterial({
            color: 0x888888,
            side: THREE.DoubleSide
        });
        const groundMesh = new THREE.Mesh(groundGeometry, groundMaterial);
        groundMesh.rotation.x = -Math.PI / 2;
        this.scene.add(groundMesh);
        
        this.meshes.set(groundBody, groundMesh);
        this.bodies.set(groundMesh, groundBody);
    }
    
    addPhysicsObject(mesh, mass = 1, shapeType = 'box') {
        let shape;
        
        switch(shapeType) {
            case 'box':
                const boxSize = mesh.geometry.boundingBox.getSize(new THREE.Vector3());
                shape = new CANNON.Box(new CANNON.Vec3(
                    boxSize.x / 2,
                    boxSize.y / 2,
                    boxSize.z / 2
                ));
                break;
            case 'sphere':
                const radius = mesh.geometry.boundingSphere.radius;
                shape = new CANNON.Sphere(radius);
                break;
            case 'cylinder':
                // Simplified cylinder physics
                shape = new CANNON.Cylinder(0.5, 0.5, 2, 8);
                break;
        }
        
        const body = new CANNON.Body({ mass: mass });
        body.addShape(shape);
        body.position.copy(mesh.position);
        body.quaternion.copy(mesh.quaternion);
        
        this.world.addBody(body);
        this.meshes.set(body, mesh);
        this.bodies.set(mesh, body);
        
        return body;
    }
    
    update(deltaTime) {
        this.world.step(deltaTime);
        
        // Update Three.js meshes from Cannon.js bodies
        for (const [body, mesh] of this.meshes) {
            mesh.position.copy(body.position);
            mesh.quaternion.copy(body.quaternion);
        }
    }
    
    // Interact with physics objects using XR controllers
    grabObject(controller, objectMesh) {
        const body = this.bodies.get(objectMesh);
        if (!body) return;
        
        // Create constraint between controller and object
        const constraint = new CANNON.PointToPointConstraint(
            body,
            new CANNON.Vec3(0, 0, 0),
            new CANNON.Body({ mass: 0 }),
            new CANNON.Vec3(0, 0, 0)
        );
        
        // Update constraint position based on controller
        controller.addEventListener('pose', (event) => {
            const pose = event.pose;
            constraint.bodyB.position.copy(pose.position);
        });
        
        this.world.addConstraint(constraint);
        
        return {
            constraint: constraint,
            release: () => {
                this.world.removeConstraint(constraint);
            }
        };
    }
}

C. Performance Optimization for WebXR

javascript

class XRPerformanceOptimizer {
    constructor(renderer) {
        this.renderer = renderer;
        this.frameTimes = [];
        this.targetFrameTime = 1000 / 90; // 90Hz target
        
        this.setupOptimizations();
        this.setupPerformanceMonitoring();
    }
    
    setupOptimizations() {
        // 1. Adaptive Quality
        this.qualityLevels = {
            low: {
                maxLights: 2,
                shadowMapSize: 512,
                antialias: false,
                textureQuality: 0.5,
                lodDistance: 10
            },
            medium: {
                maxLights: 4,
                shadowMapSize: 1024,
                antialias: true,
                textureQuality: 0.75,
                lodDistance: 20
            },
            high: {
                maxLights: 8,
                shadowMapSize: 2048,
                antialias: true,
                textureQuality: 1.0,
                lodDistance: 50
            }
        };
        
        this.currentQuality = 'high';
        
        // 2. Dynamic Resolution Scaling
        this.baseResolution = 1.0;
        this.minResolution = 0.5;
        this.maxResolution = 1.5;
        this.resolutionScale = 1.0;
        
        // 3. Frustum Culling Optimization
        this.renderer.localClippingEnabled = true;
        
        // 4. Instanced Rendering
        this.instancedMeshes = new Map();
    }
    
    setupPerformanceMonitoring() {
        // Use WebXR's frame rate monitoring
        this.renderer.xr.addEventListener('sessionstart', (event) => {
            const session = event.session;
            
            // Request frame rate
            session.updateTargetFrameRate(90).catch(() => {
                console.log('90Hz not supported, falling back to default');
            });
            
            // Monitor performance
            this.startPerformanceMonitoring();
        });
    }
    
    startPerformanceMonitoring() {
        let lastTime = performance.now();
        
        const monitorFrame = () => {
            const currentTime = performance.now();
            const frameTime = currentTime - lastTime;
            lastTime = currentTime;
            
            // Store frame time
            this.frameTimes.push(frameTime);
            if (this.frameTimes.length > 60) {
                this.frameTimes.shift();
            }
            
            // Calculate average frame time
            const avgFrameTime = this.frameTimes.reduce((a, b) => a + b) / this.frameTimes.length;
            
            // Adjust quality based on performance
            this.adjustQuality(avgFrameTime);
            
            // Adjust resolution scaling
            this.adjustResolutionScaling(avgFrameTime);
            
            requestAnimationFrame(monitorFrame);
        };
        
        monitorFrame();
    }
    
    adjustQuality(avgFrameTime) {
        const targetFPS = 90;
        const targetFrameTime = 1000 / targetFPS;
        const safetyMargin = 1.1; // 10% safety margin
        
        if (avgFrameTime > targetFrameTime * safetyMargin) {
            // Drop quality level
            if (this.currentQuality === 'high') {
                this.setQuality('medium');
            } else if (this.currentQuality === 'medium') {
                this.setQuality('low');
            }
        } else if (avgFrameTime < targetFrameTime / safetyMargin) {
            // Increase quality level
            if (this.currentQuality === 'low') {
                this.setQuality('medium');
            } else if (this.currentQuality === 'medium') {
                this.setQuality('high');
            }
        }
    }
    
    setQuality(level) {
        if (this.currentQuality === level) return;
        
        this.currentQuality = level;
        const settings = this.qualityLevels[level];
        
        // Apply quality settings
        this.renderer.shadowMap.enabled = level !== 'low';
        this.renderer.shadowMap.type = level === 'high' ? 
            THREE.PCFSoftShadowMap : THREE.BasicShadowMap;
        this.renderer.shadowMap.size.width = settings.shadowMapSize;
        this.renderer.shadowMap.size.height = settings.shadowMapSize;
        
        // Update scene LOD distances
        this.scene.traverse((object) => {
            if (object.isLOD) {
                object.levels.forEach((levelObj, i) => {
                    levelObj.distance *= (settings.lodDistance / 50);
                });
            }
        });
        
        console.log(`Quality set to: ${level}`);
    }
    
    adjustResolutionScaling(avgFrameTime) {
        const targetFrameTime = this.targetFrameTime;
        const currentScale = this.resolutionScale;
        
        if (avgFrameTime > targetFrameTime * 1.2) {
            // Reduce resolution
            this.resolutionScale = Math.max(
                this.minResolution,
                currentScale * 0.95
            );
        } else if (avgFrameTime < targetFrameTime * 0.8) {
            // Increase resolution
            this.resolutionScale = Math.min(
                this.maxResolution,
                currentScale * 1.05
            );
        }
        
        // Apply resolution scaling
        this.renderer.setPixelRatio(window.devicePixelRatio * this.resolutionScale);
    }
    
    // Instanced rendering for many similar objects
    createInstancedMesh(geometry, material, count) {
        const instancedMesh = new THREE.InstancedMesh(geometry, material, count);
        instancedMesh.instanceMatrix.setUsage(THREE.DynamicDrawUsage);
        
        this.instancedMeshes.set(instancedMesh, {
            count: count,
            matrices: new Array(count).fill().map(() => new THREE.Matrix4()),
            colors: new Array(count).fill().map(() => new THREE.Color())
        });
        
        return instancedMesh;
    }
    
    updateInstancedMesh(mesh, instanceIndex, position, rotation, scale, color) {
        const data = this.instancedMeshes.get(mesh);
        if (!data || instanceIndex >= data.count) return;
        
        const matrix = data.matrices[instanceIndex];
        matrix.compose(position, rotation, scale);
        
        mesh.setMatrixAt(instanceIndex, matrix);
        
        if (color && mesh.instanceColor) {
            mesh.setColorAt(instanceIndex, color);
        }
        
        mesh.instanceMatrix.needsUpdate = true;
        if (mesh.instanceColor) {
            mesh.instanceColor.needsUpdate = true;
        }
    }
    
    // Occlusion culling with custom implementation
    setupOcclusionCulling(scene, camera) {
        const cullingHelper = new THREE.Box3Helper(new THREE.Box3(), 0x00ff00);
        scene.add(cullingHelper);
        
        const frustum = new THREE.Frustum();
        const projScreenMatrix = new THREE.Matrix4();
        
        return {
            update: () => {
                projScreenMatrix.multiplyMatrices(
                    camera.projectionMatrix,
                    camera.matrixWorldInverse
                );
                frustum.setFromProjectionMatrix(projScreenMatrix);
                
                let visibleCount = 0;
                let totalCount = 0;
                
                scene.traverse((object) => {
                    if (object.isMesh && object.geometry) {
                        totalCount++;
                        
                        if (object.geometry.boundingBox) {
                            if (frustum.intersectsBox(object.geometry.boundingBox)) {
                                object.visible = true;
                                visibleCount++;
                            } else {
                                object.visible = false;
                            }
                        }
                    }
                });
                
                return { visible: visibleCount, total: totalCount };
            }
        };
    }
}

3. Unity 6 for XR Development

A. Unity 6 XR Architecture & Setup

csharp

// Unity 6 XR Manager Script
using UnityEngine;
using UnityEngine.XR;
using UnityEngine.XR.Management;
using System.Collections;
using System.Collections.Generic;

public class XRManagerUnity6 : MonoBehaviour
{
    [Header("XR Settings")]
    [SerializeField] private XRLoaderType loaderType = XRLoaderType.OpenXR;
    [SerializeField] private float renderScale = 1.0f;
    [SerializeField] private int targetFPS = 90;
    [SerializeField] private bool enableFoveatedRendering = true;
    
    [Header("Quality Settings")]
    [SerializeField] private AdaptiveQuality adaptiveQuality;
    [SerializeField] private bool enableMSAA = true;
    [SerializeField] private ShadowQuality shadowQuality = ShadowQuality.High;
    
    private XRLoader xrLoader;
    private Camera xrCamera;
    private List<InputDevice> inputDevices = new List<InputDevice>();
    
    public enum XRLoaderType
    {
        OpenXR,
        Oculus,
        WindowsMR,
        ARFoundation
    }
    
    public enum ShadowQuality
    {
        Off,
        Low,
        Medium,
        High
    }
    
    [System.Serializable]
    public class AdaptiveQuality
    {
        public bool enabled = true;
        public float minResolution = 0.7f;
        public float maxResolution = 1.3f;
        public int targetFrameTimeMS = 11; // ~90Hz
        public float adjustmentSpeed = 0.1f;
    }
    
    void Start()
    {
        InitializeXR();
        SetupCamera();
        SetupInput();
        SetupPerformanceOptimizations();
    }
    
    async void InitializeXR()
    {
        // Initialize XR subsystem
        var xrSettings = XRGeneralSettings.Instance;
        if (xrSettings == null)
        {
            Debug.LogError("XR General Settings not found!");
            return;
        }
        
        var xrManager = xrSettings.Manager;
        
        // Stop any existing loader
        if (xrManager.activeLoader != null)
        {
            xrManager.DeinitializeLoader();
        }
        
        // Create and initialize loader based on type
        switch (loaderType)
        {
            case XRLoaderType.OpenXR:
                xrLoader = gameObject.AddComponent<UnityEngine.XR.OpenXR.OpenXRLoader>();
                break;
            case XRLoaderType.Oculus:
                xrLoader = gameObject.AddComponent<Unity.XR.Oculus.OculusLoader>();
                break;
            case XRLoaderType.WindowsMR:
                xrLoader = gameObject.AddComponent<UnityEngine.XR.WindowsMR.WindowsMRLoader>();
                break;
            case XRLoaderType.ARFoundation:
                await InitializeARFoundation();
                return;
        }
        
        // Initialize loader
        if (xrLoader.Initialize())
        {
            Debug.Log($"XR Loader initialized: {loaderType}");
            
            // Start subsystems
            xrLoader.Start();
            
            // Setup XR display
            XRSettings.eyeTextureResolutionScale = renderScale;
            Application.targetFrameRate = targetFPS;
            
            Debug.Log("XR initialization complete");
        }
        else
        {
            Debug.LogError($"Failed to initialize XR loader: {loaderType}");
        }
    }
    
    async Task InitializeARFoundation()
    {
        // AR Foundation setup for mobile AR
        var arSession = FindObjectOfType<ARSession>();
        if (arSession == null)
        {
            arSession = new GameObject("AR Session").AddComponent<ARSession>();
        }
        
        var arSessionOrigin = FindObjectOfType<ARSessionOrigin>();
        if (arSessionOrigin == null)
        {
            arSessionOrigin = new GameObject("AR Session Origin").AddComponent<ARSessionOrigin>();
        }
        
        // Setup AR subsystems
        var planeManager = arSessionOrigin.gameObject.AddComponent<ARPlaneManager>();
        var raycastManager = arSessionOrigin.gameObject.AddComponent<ARRaycastManager>();
        var anchorManager = arSessionOrigin.gameObject.AddComponent<ARAnchorManager>();
        
        // Configure plane detection
        planeManager.requestedDetectionMode = PlaneDetectionMode.Horizontal;
        planeManager.planePrefab = Resources.Load<GameObject>("ARPlanePrefab");
        
        Debug.Log("AR Foundation initialized");
    }
    
    void SetupCamera()
    {
        // Get or create XR camera
        xrCamera = Camera.main;
        if (xrCamera == null)
        {
            xrCamera = new GameObject("XR Camera").AddComponent<Camera>();
        }
        
        // Add XR components
        var cameraOffset = xrCamera.gameObject.AddComponent<UnityEngine.XR.Interaction.Toolkit.XROrigin>();
        var cameraFollow = xrCamera.gameObject.AddComponent<UnityEngine.XR.Interaction.Toolkit.CameraFollow>();
        
        // Configure camera
        xrCamera.nearClipPlane = 0.1f;
        xrCamera.farClipPlane = 1000f;
        xrCamera.clearFlags = CameraClearFlags.Skybox;
        
        // Add post-processing for XR
        SetupPostProcessing();
    }
    
    void SetupPostProcessing()
    {
        // Add XR-optimized post-processing
        var postProcessLayer = xrCamera.gameObject.AddComponent<UnityEngine.Rendering.PostProcessing.PostProcessLayer>();
        postProcessLayer.antialiasingMode = UnityEngine.Rendering.PostProcessing.PostProcessLayer.Antialiasing.FastApproximateAntialiasing;
        postProcessLayer.fastApproximateAntialiasing.fastMode = true;
        
        // Chromatic aberration correction for lenses
        var chromaticVolume = xrCamera.gameObject.AddComponent<UnityEngine.Rendering.PostProcessing.PostProcessVolume>();
        chromaticVolume.isGlobal = true;
        var chromaticProfile = ScriptableObject.CreateInstance<UnityEngine.Rendering.PostProcessing.PostProcessProfile>();
        var chromaticAberration = chromaticProfile.AddSettings<UnityEngine.Rendering.PostProcessing.ChromaticAberration>();
        chromaticAberration.intensity.value = 0.1f;
        chromaticVolume.profile = chromaticProfile;
    }
    
    void SetupInput()
    {
        // Setup XR input devices
        InputDevices.deviceConnected += OnDeviceConnected;
        InputDevices.deviceDisconnected += OnDeviceDisconnected;
        
        // Get initial devices
        InputDevices.GetDevices(inputDevices);
        
        // Setup action-based input system
        SetupActionBasedInput();
    }
    
    void SetupActionBasedInput()
    {
        var inputManager = gameObject.AddComponent<UnityEngine.XR.Interaction.Toolkit.ActionBasedControllerManager>();
        
        // Left controller
        var leftHand = new GameObject("Left Hand Controller").AddComponent<UnityEngine.XR.Interaction.Toolkit.ActionBasedController>();
        leftHand.controllerNode = UnityEngine.XR.XRNode.LeftHand;
        
        // Right controller
        var rightHand = new GameObject("Right Hand Controller").AddComponent<UnityEngine.XR.Interaction.Toolkit.ActionBasedController>();
        rightHand.controllerNode = UnityEngine.XR.XRNode.RightHand;
        
        // Setup common input actions
        var teleportAction = new UnityEngine.InputSystem.InputAction("Teleport", InputActionType.Button);
        var grabAction = new UnityEngine.InputSystem.InputAction("Grab", InputActionType.Button);
        var uiInteractAction = new UnityEngine.InputSystem.InputAction("UI Interact", InputActionType.Button);
        
        // Assign actions to controllers
        leftHand.teleportSelectAction = teleportAction;
        rightHand.teleportSelectAction = teleportAction;
        leftHand.selectAction = grabAction;
        rightHand.selectAction = grabAction;
        leftHand.uiPressAction = uiInteractAction;
        rightHand.uiPressAction = uiInteractAction;
    }
    
    void SetupPerformanceOptimizations()
    {
        // Configure Unity quality settings for XR
        QualitySettings.vSyncCount = 0;
        QualitySettings.maxQueuedFrames = 1;
        QualitySettings.shadowDistance = 20f;
        QualitySettings.shadowCascade = 2;
        
        // Configure rendering for XR
        SetupXRendering();
        
        // Setup adaptive quality
        if (adaptiveQuality.enabled)
        {
            StartCoroutine(AdaptiveQualityRoutine());
        }
        
        // Setup foveated rendering if supported
        if (enableFoveatedRendering)
        {
            SetupFoveatedRendering();
        }
    }
    
    void SetupXRendering()
    {
        // Single Pass Instanced rendering for VR
        XRSettings.eyeTextureResolutionScale = renderScale;
        XRSettings.renderViewportScale = 1.0f;
        
        // Configure shadows based on quality setting
        switch (shadowQuality)
        {
            case ShadowQuality.Off:
                QualitySettings.shadows = ShadowQuality.Disable;
                break;
            case ShadowQuality.Low:
                QualitySettings.shadows = ShadowQuality.HardOnly;
                QualitySettings.shadowResolution = ShadowResolution.Low;
                break;
            case ShadowQuality.Medium:
                QualitySettings.shadows = ShadowQuality.All;
                QualitySettings.shadowResolution = ShadowResolution.Medium;
                break;
            case ShadowQuality.High:
                QualitySettings.shadows = ShadowQuality.All;
                QualitySettings.shadowResolution = ShadowResolution.High;
                QualitySettings.shadowCascades = 4;
                break;
        }
        
        // Configure MSAA
        QualitySettings.antiAliasing = enableMSAA ? 4 : 0;
    }
    
    void SetupFoveatedRendering()
    {
        // Check if foveated rendering is supported
        if (SystemInfo.supportsFoveatedRendering)
        {
            // Enable fixed foveated rendering
            XRSettings.foveatedRendering = true;
            XRSettings.foveatedRenderingLevel = FoveatedRenderingLevel.High;
            
            Debug.Log("Foveated rendering enabled");
        }
        else
        {
            Debug.LogWarning("Foveated rendering not supported on this platform");
        }
    }
    
    IEnumerator AdaptiveQualityRoutine()
    {
        float currentScale = renderScale;
        
        while (true)
        {
            yield return new WaitForSeconds(0.5f); // Check every 0.5 seconds
            
            // Calculate current frame time
            float frameTime = Time.unscaledDeltaTime * 1000f; // Convert to ms
            
            // Adjust resolution scale based on performance
            if (frameTime > adaptiveQuality.targetFrameTimeMS)
            {
                // Performance is bad, reduce resolution
                currentScale = Mathf.Max(
                    adaptiveQuality.minResolution,
                    currentScale - adaptiveQuality.adjustmentSpeed
                );
            }
            else if (frameTime < adaptiveQuality.targetFrameTimeMS * 0.8f)
            {
                // Performance is good, increase resolution
                currentScale = Mathf.Min(
                    adaptiveQuality.maxResolution,
                    currentScale + adaptiveQuality.adjustmentSpeed
                );
            }
            
            // Apply resolution scale
            XRSettings.eyeTextureResolutionScale = currentScale;
            
            Debug.Log($"Adaptive Quality: Scale={currentScale:F2}, FrameTime={frameTime:F2}ms");
        }
    }
    
    void OnDeviceConnected(InputDevice device)
    {
        if (!inputDevices.Contains(device))
        {
            inputDevices.Add(device);
            Debug.Log($"Device connected: {device.name}");
            
            // Setup device-specific features
            SetupDeviceFeatures(device);
        }
    }
    
    void SetupDeviceFeatures(InputDevice device)
    {
        // Check for specific features
        if (device.characteristics.HasFlag(InputDeviceCharacteristics.HandTracking))
        {
            SetupHandTracking(device);
        }
        
        if (device.characteristics.HasFlag(InputDeviceCharacteristics.EyeTracking))
        {
            SetupEyeTracking(device);
        }
        
        if (device.characteristics.HasFlag(InputDeviceCharacteristics.Haptic))
        {
            SetupHaptics(device);
        }
    }
    
    void SetupHandTracking(InputDevice device)
    {
        Debug.Log($"Setting up hand tracking for {device.name}");
        
        // Create hand visualizations
        var handVisualizer = gameObject.AddComponent<XRHandVisualizer>();
        handVisualizer.Initialize(device);
    }
    
    void SetupEyeTracking(InputDevice device)
    {
        Debug.Log($"Setting up eye tracking for {device.name}");
        
        // Enable dynamic foveated rendering if supported
        if (SystemInfo.supportsEyeTracking)
        {
            XRSettings.foveatedRenderingCaps = FoveatedRenderingCaps.EyeTracked;
            Debug.Log("Dynamic foveated rendering enabled");
        }
    }
    
    void SetupHaptics(InputDevice device)
    {
        // Setup haptic feedback patterns
        var hapticManager = gameObject.GetComponent<XRHapticManager>();
        if (hapticManager == null)
        {
            hapticManager = gameObject.AddComponent<XRHapticManager>();
        }
        
        hapticManager.RegisterDevice(device);
    }
    
    void OnDeviceDisconnected(InputDevice device)
    {
        inputDevices.Remove(device);
        Debug.Log($"Device disconnected: {device.name}");
    }
    
    void Update()
    {
        // Update input devices
        foreach (var device in inputDevices)
        {
            UpdateDeviceInput(device);
        }
        
        // Update performance monitoring
        UpdatePerformanceMonitoring();
    }
    
    void UpdateDeviceInput(InputDevice device)
    {
        // Read controller inputs
        device.TryGetFeatureValue(CommonUsages.trigger, out float triggerValue);
        device.TryGetFeatureValue(CommonUsages.grip, out float gripValue);
        device.TryGetFeatureValue(CommonUsages.primaryButton, out bool primaryButton);
        device.TryGetFeatureValue(CommonUsages.secondaryButton, out bool secondaryButton);
        
        // Process input
        if (triggerValue > 0.1f)
        {
            OnTriggerPressed(device, triggerValue);
        }
        
        if (gripValue > 0.1f)
        {
            OnGripPressed(device, gripValue);
        }
    }
    
    void UpdatePerformanceMonitoring()
    {
        // Monitor GPU and CPU time
        float gpuTime = 0f;
        float cpuTime = 0f;
        
        // Unity 6 provides better performance counters
        #if UNITY_2022_3_OR_NEWER
        gpuTime = UnityEngine.Rendering.RenderPipelineManager.currentPipeline?.GetGPUFrameTime() ?? 0f;
        cpuTime = UnityEngine.Rendering.RenderPipelineManager.currentPipeline?.GetCPUFrameTime() ?? 0f;
        #endif
        
        // Log performance issues
        if (gpuTime > 1000f / targetFPS)
        {
            Debug.LogWarning($"GPU bound: {gpuTime:F2}ms");
        }
        
        if (cpuTime > 1000f / targetFPS)
        {
            Debug.LogWarning($"CPU bound: {cpuTime:F2}ms");
        }
    }
    
    void OnTriggerPressed(InputDevice device, float value)
    {
        // Handle trigger press
        // Could be shooting, grabbing, etc.
    }
    
    void OnGripPressed(InputDevice device, float value)
    {
        // Handle grip press
        // Could be holding objects, menu access, etc.
    }
    
    void OnDestroy()
    {
        // Cleanup
        if (xrLoader != null)
        {
            xrLoader.Stop();
            xrLoader.Deinitialize();
        }
        
        InputDevices.deviceConnected -= OnDeviceConnected;
        InputDevices.deviceDisconnected -= OnDeviceDisconnected;
    }
}